Hi all, I'm having an issue with avformat_find_stream_info() not pulling the timing information out of an H264 stream that I am parsing in real time as it comes off of my camera. I've cobbled together my stream detection code by reading scattered tutorials, libav-user threads, and diving into the libavformat/libavcodec code base, so there is much room for error on my part in how I am trying to achieve this. That said, I think it is prudent that I start by explaining my use case thoroughly for context, following which I will provide my code and results.
I have a custom camera built by an OEM which outputs a 1080p@30fps H264 elementary stream over USB. Its interface is simple: I provide it a callback that takes a buffer, a size, and void* userData, and it calls that callback each time it has finished producing a frame. The frame is guaranteed to be a complete NAL unit. My goal is to take the live stream from the camera, detect the format of the video, mux each frame into a fragmented MP4 container (1 NAL unit per container), and shoot each muxed frame off to another program to be used as a live stream. I am also trying to get each of these frames through the pipeline with the lowest latency possible, trying to avoid buffering of any form. For the most part, this process is working great and I've got live MP4 video streaming in my other application and rendering correctly. However, in the stream format detection stage, I get some warning messages and lots of missing fields when it comes to timing information, and that missing timing information sometimes causes problems in my other program (and other video services) when it comes to playback speed. All of my re-muxing logic is contained in this class: https://github.com/OpenROV-Dev/geomuxpp/blob/inproc_rewrite/src/CMuxer.h https://github.com/OpenROV-Dev/geomuxpp/blob/inproc_rewrite/src/CMuxer.cpp Data is dumped into the CVideoBuffer m_inputBuffer structure by the camera callback. When data is available, the muxer gets signaled to process the NAL unit. First it builds up a couple megabytes of data to probe, then it finds the stream info, and ultimately constructs an output AVIO context for muxing to MP4 using all of the parsed stream info. Then each packet is muxed and written out using the custom WritePacket callback which sends the data out over ZeroMQ. Here is the output of my program with AV traces on (Note: Using FFMPEG latest stable source tarball, 3.0.1 from 2/24/16): ------------------------- ---------NEW RUN--------- Channels created: 1 Application Started. Init success Input: 21092 bytes at: 1461658579098 Input: 45508 bytes at: 1461658579132 Input: 39348 bytes at: 1461658579165 Input: 37716 bytes at: 1461658579199 Input: 38372 bytes at: 1461658579232 Input: 37016 bytes at: 1461658579265 Input: 37324 bytes at: 1461658579299 Input: 35600 bytes at: 1461658579332 Input: 35400 bytes at: 1461658579365 Input: 36160 bytes at: 1461658579399 Input: 36708 bytes at: 1461658579432 Input: 36472 bytes at: 1461658579465 Input: 36896 bytes at: 1461658579499 Input: 34628 bytes at: 1461658579532 Input: 34728 bytes at: 1461658579565 Input: 34476 bytes at: 1461658579598 Input: 35084 bytes at: 1461658579632 Input: 36344 bytes at: 1461658579665 Input: 36200 bytes at: 1461658579698 Input: 38156 bytes at: 1461658579732 Input: 35280 bytes at: 1461658579765 Input: 34252 bytes at: 1461658579798 Input: 34768 bytes at: 1461658579832 Input: 34612 bytes at: 1461658579865 Input: 35624 bytes at: 1461658579898 Input: 36408 bytes at: 1461658579932 Input: 35048 bytes at: 1461658579965 Input: 35252 bytes at: 1461658579998 Input: 34852 bytes at: 1461658580032 Input: 33968 bytes at: 1461658580065 Input: 34004 bytes at: 1461658580098 Input: 35048 bytes at: 1461658580132 Input: 34676 bytes at: 1461658580165 Input: 35212 bytes at: 1461658580198 Input: 34636 bytes at: 1461658580232 Input: 33448 bytes at: 1461658580265 Input: 34016 bytes at: 1461658580298 Input: 34288 bytes at: 1461658580332 Input: 34244 bytes at: 1461658580365 Input: 34276 bytes at: 1461658580398 Input: 34628 bytes at: 1461658580431 Input: 34212 bytes at: 1461658580465 Input: 33684 bytes at: 1461658580498 Input: 33536 bytes at: 1461658580531 Input: 33520 bytes at: 1461658580565 Input: 35332 bytes at: 1461658580598 Input: 34404 bytes at: 1461658580631 Input: 33956 bytes at: 1461658580665 Input: 33176 bytes at: 1461658580698 Input: 34100 bytes at: 1461658580731 Input: 34072 bytes at: 1461658580765 Input: 33724 bytes at: 1461658580798 Input: 34064 bytes at: 1461658580831 Input: 33976 bytes at: 1461658580865 Input: 33888 bytes at: 1461658580898 Input: 34124 bytes at: 1461658580931 Input: 34008 bytes at: 1461658580965 Input: 33412 bytes at: 1461658580998 Probing input buffer for format info... Input: 33892 bytes at: 1461658581031 Input: 33312 bytes at: 1461658581065 Input: 33012 bytes at: 1461658581098 Input: 33724 bytes at: 1461658581131 Input: 33900 bytes at: 1461658581165 Input: 34272 bytes at: 1461658581198 Input: 33016 bytes at: 1461658581231 Input: 33612 bytes at: 1461658581265 Input: 33300 bytes at: 1461658581298 Input: 34348 bytes at: 1461658581331 Input: 34340 bytes at: 1461658581365 Input: 34268 bytes at: 1461658581398 Input: 33764 bytes at: 1461658581431 Input: 34008 bytes at: 1461658581464 Input: 33800 bytes at: 1461658581498 Input: 34504 bytes at: 1461658581531 Input: 33864 bytes at: 1461658581564 Input: 34600 bytes at: 1461658581598 Input: 34528 bytes at: 1461658581631 Input: 34516 bytes at: 1461658581664 Input: 35412 bytes at: 1461658581698 Input: 34676 bytes at: 1461658581731 Input: 34936 bytes at: 1461658581764 Input: 34548 bytes at: 1461658581798 Input: 34156 bytes at: 1461658581831 Input: 34256 bytes at: 1461658581864 Input: 33472 bytes at: 1461658581898 Input: 34264 bytes at: 1461658581931 Input: 33808 bytes at: 1461658581964 Input: 33868 bytes at: 1461658581998 Input: 101692 bytes at: 1461658582029 Input: 13188 bytes at: 1461658582064 Input: 244 bytes at: 1461658582097 Input: 936 bytes at: 1461658582130 Input: 5140 bytes at: 1461658582164 Input: 16340 bytes at: 1461658582197 Input: 43772 bytes at: 1461658582231 Input: 38212 bytes at: 1461658582264 Input: 36572 bytes at: 1461658582298 Input: 36184 bytes at: 1461658582331 Input: 36608 bytes at: 1461658582364 Input: 35452 bytes at: 1461658582398 Input: 35320 bytes at: 1461658582431 Input: 35088 bytes at: 1461658582464 Input: 34248 bytes at: 1461658582497 Input: 35396 bytes at: 1461658582531 Input: 35016 bytes at: 1461658582564 Input: 35368 bytes at: 1461658582597 Input: 34368 bytes at: 1461658582631 Input: 34492 bytes at: 1461658582664 Input: 35032 bytes at: 1461658582697 Input: 35036 bytes at: 1461658582731 Input: 34364 bytes at: 1461658582764 Input: 33140 bytes at: 1461658582797 Input: 33800 bytes at: 1461658582831 Input: 33888 bytes at: 1461658582864 Input: 33548 bytes at: 1461658582897 Input: 33196 bytes at: 1461658582931 Input: 33556 bytes at: 1461658582964 Input: 34600 bytes at: 1461658582997 Probing input buffer for format info... Probing h264 score:51 size:2005772 Successfully acquired input format. Input opened successfully! Finding stream info... [h264 @ 0x7fcd400008c0] Before avformat_find_stream_info() pos: 0 bytes read:2005772 seeks:0 [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced Input: [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! 34228 bytes at: 1461658583031 [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] non-existing PPS 0 referenced [h264 @ 0x7fcd400022a0] decode_slice_header error [h264 @ 0x7fcd400022a0] no frame! [h264 @ 0x7fcd400022a0] Current profile doesn't provide more RBSP data in PPS, skipping Input: 33400 bytes at: 1461658583064 [h264 @ 0x7fcd400008c0] 0: start_time: -9223372036854.775 duration: -9223372036854.775 [h264 @ 0x7fcd400008c0] stream: start_time: -9223372036854.775 duration: -9223372036854.775 bitrate=0 kb/s [h264 @ 0x7fcd400008c0] After avformat_find_stream_info() pos: 2073400 bytes read:2073400 seeks:0 frames:62 Input #0, h264, from '(null)': Duration: N/A, bitrate: N/A Stream #0:0, 62, 1/1200000: Video: h264 (Main), 1 reference frame, yuv420p(tv, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 1001/30000, 17.25 fps, 14.99 tbr, 1200k tbn, 29.97 tbc Stream info acquired. Finding codec... Opening codec... detected 8 logical cores Codec opened. Created output stream. Copied input codec context to output codec context. Writing FTYP+MOOV... [mp4 @ 0x7fcd400010c0] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead. Writing muxed packet. Bytes: 24 Writing muxed packet. Bytes: 754 Ready to mux! Input: 33764 bytes at: 1461658583097 [mp4 @ 0x7fcd400010c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 0x7fcd400010c0] Encoder did not produce proper pts, making some up. Writing muxed packet. Bytes: 33872 Input: 33804 bytes at: 1461658583131 Writing muxed packet. Bytes: 33912 Input: 34424 bytes at: 1461658583164 Writing muxed packet. Bytes: 34532 Input: 34152 bytes at: 1461658583197 Writing muxed packet. Bytes: 34260 Input: 34168 bytes at: 1461658583231 Writing muxed packet. Bytes: 34276 Input: 34072 bytes at: 1461658583264 Writing muxed packet. Bytes: 34180 Input: 33940 bytes at: 1461658583297 Writing muxed packet. Bytes: 34048 Input: 34448 bytes at: 1461658583331 Writing muxed packet. Bytes: 34556 Input: 33808 bytes at: 1461658583364 Writing muxed packet. Bytes: 33916 Input: 33916 bytes at: 1461658583397 Writing muxed packet. Bytes: 34024 Input: 34492 bytes at: 1461658583431 Writing muxed packet. Bytes: 34600 Input: 34384 bytes at: 1461658583464 Writing muxed packet. Bytes: 34492 Input: 33324 bytes at: 1461658583497 Writing muxed packet. Bytes: 33432 Input: 33184 bytes at: 1461658583530 Writing muxed packet. Bytes: 33292 Input: 33336 bytes at: 1461658583564 Writing muxed packet. Bytes: 33444 Input: 32800 bytes at: 1461658583597 Writing muxed packet. Bytes: 32908 Input: 32956 bytes at: 1461658583630 Writing muxed packet. Bytes: 33064 Input: 33092 bytes at: 1461658583664 Writing muxed packet. Bytes: 33200 Input: 32972 bytes at: 1461658583697 Writing muxed packet. Bytes: 33080 Input: 33516 bytes at: 1461658583730 Writing muxed packet. Bytes: 33624 Input: 33700 bytes at: 1461658583764 Writing muxed packet. Bytes: 33808 Input: 33836 bytes at: 1461658583797 Writing muxed packet. Bytes: 33944 Input: 34164 bytes at: 1461658583830 Writing muxed packet. Bytes: 34272 Input: 34460 bytes at: 1461658583864 Writing muxed packet. Bytes: 34568 Input: 34052 bytes at: 1461658583897 Writing muxed packet. Bytes: 34160 Input: 34072 bytes at: 1461658583930 Writing muxed packet. Bytes: 34180 Input: 34720 bytes at: 1461658583964 Writing muxed packet. Bytes: 34828 ^C Breaking this down, we save up a bunch of packets, attempt a probe, but it fails. We save up some more frames and probe again. This time, the first 30 or so frames in the probe data buffer were associated with a PPS/SPS in the previous discarded probeData buffer, so I believe that explains the first stream of error messages. Eventually, it gets a new IDR-frame with a fresh PPS/SPS and the probe is successful. The next bit brings me to my first question: [h264 @ 0x7fcd400008c0] 0: start_time: -9223372036854.775 duration: -9223372036854.775 [h264 @ 0x7fcd400008c0] stream: start_time: -9223372036854.775 duration: -9223372036854.775 bitrate=0 kb/s [h264 @ 0x7fcd400008c0] After avformat_find_stream_info() pos: 2073400 bytes read:2073400 seeks:0 frames:62 Why are these time stamps and bitrate always garbage? Calling av_dump_format on the input format context gives: Input #0, h264, from '(null)': Duration: N/A, bitrate: N/A Stream #0:0, 62, 1/1200000: Video: h264 (Main), 1 reference frame, yuv420p(tv, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 1001/30000, 17.25 fps, 14.99 tbr, 1200k tbn, 29.97 tbc The FPS value here is weird, as it is a 30fps stream, as evidenced by my callback timestamps (units in milliseconds). The TBC looks correct, but the fps is way off, and the TBR is half of what I expect it to be. No idea where the 1200k TBN came from. Here is the data contained in an SPS segment of this stream. There is some timing information under the VUI section: ==================== NAL ==================== forbidden_zero_bit :0 nal_ref_idc :3 nal_unit_type : 7Sequence parameter set ======= SPS ======= profile_idc :77 constraint_set0_flag :0 constraint_set1_flag :1 constraint_set2_flag :0 constraint_set3_flag :0 constraint_set4_flag :0 constraint_set5_flag :0 reserved_zero_2bits :0 level_idc :31 seq_parameter_set_id :0 chroma_format_idc :1 residual_colour_transform_flag :0 bit_depth_luma_minus8 :0 bit_depth_chroma_minus8 :0 qpprime_y_zero_transform_bypass_flag :0 seq_scaling_matrix_present_flag :0 log2_max_frame_num_minus4 :0 pic_order_cnt_type :0 log2_max_pic_order_cnt_lsb_minus4 :6 delta_pic_order_always_zero_flag :0 offset_for_non_ref_pic :0 offset_for_top_to_bottom_field :0 num_ref_frames_in_pic_order_cnt_cycle :0 num_ref_frames :1 gaps_in_frame_num_value_allowed_flag :0 pic_width_in_mbs_minus1 :119 pic_height_in_map_units_minus1 :67 frame_mbs_only_flag :1 mb_adaptive_frame_field_flag :0 direct_8x8_inference_flag :1 frame_cropping_flag :1 frame_crop_left_offset :0 frame_crop_right_offset :0 frame_crop_top_offset :0 frame_crop_bottom_offset :4 vui_parameters_present_flag :1 === VUI === aspect_ratio_info_present_flag :1 aspect_ratio_idc :1 sar_width :0 sar_height :0 overscan_info_present_flag :0 overscan_appropriate_flag :0 video_signal_type_present_flag :1 video_format :0 video_full_range_flag :0 colour_description_present_flag :0 colour_primaries :0 transfer_characteristics :0 matrix_coefficients :0 chroma_loc_info_present_flag :0 chroma_sample_loc_type_top_field :0 chroma_sample_loc_type_bottom_field :0 timing_info_present_flag :1 num_units_in_tick :6006 time_scale :180000 fixed_frame_rate_flag :0 nal_hrd_parameters_present_flag :1 vcl_hrd_parameters_present_flag :1 low_delay_hrd_flag :0 pic_struct_present_flag :0 bitstream_restriction_flag :0 motion_vectors_over_pic_boundaries_flag :0 max_bytes_per_pic_denom :0 max_bits_per_mb_denom :0 log2_max_mv_length_horizontal :0 log2_max_mv_length_vertical :0 num_reorder_frames :0 max_dec_frame_buffering :0 === HRD === cpb_cnt_minus1 :0 bit_rate_scale :0 cpb_size_scale :0 bit_rate_value_minus1[0] :218749 cpb_size_value_minus1[0] :874999 cbr_flag[0] :0 initial_cpb_removal_delay_length_minus1 :31 cpb_removal_delay_length_minus1 :31 dpb_output_delay_length_minus1 :31 time_offset_length :24 The encoder goes on to give me a few more warnings, which I can only assume are related to having bad timing info from the decode step. In summary, does anyone know why this timing info might be so botched? Additionally, is the logic employed in my code sound? There is very little in the way of documentation/tutorials for working with raw streams from a custom input buffer, and I haven't really found anyone else trying to mux each frame like I have, so it is very possible that I've made many mistakes (I'm surprised this works as well as it does). Thanks, Charles
_______________________________________________ Libav-user mailing list [email protected] http://ffmpeg.org/mailman/listinfo/libav-user
