Further to this, did some more debugging. See below for the 1-line fix. To clarify the situation: I determined that my dropbox was running with --normalization-level=0 due to a configuration mistake not a bug, but this should be a valid setting; it should mean simply "do not normalize dropped files". However, even in this situation the RD code alters the format of the imported files enough times to cause loss of FP precision and integer overflow, leading to distortion. The path goes like this:
rdimport makes a web services call against web/rdxport.cgi, which eventually ends up in lib/rdaudioconvert.cpp. Here the fun starts. Stage 1: we try to decode the input file to a WAV. My input is an MP3 so we use libmad. The output file, despite being named signed1 is actually a float32 formatted WAV. We explicitly disable float normalisation (mapping integer types to the range [-1, 1]), but this doesn't matter as we only write it using sf_writef_float calls. Potential causes of trouble here: MAD gives us doubles, which we cast to single-precision floats. Potentially fatal loss of precision. Stage 2: we open the stage1 output WAV; again we explicitly disable float normalisation and again it doesn't matter as that setting only affects reads from integer formatted files. This stage *would* apply transformations like normalization (gain), sample rate conversion, speed alterations etc, except I have none configured, so it's actually a tight sf_readf_float / sf_writef_float loop. The critical thing is that the output file here, named signed2, actually *is* a 32-bit signed integer WAV. By default libsndfile does not checlk whether floats passed to sf_writef_float are within the valid [-1, 1] range; it simply multiplies its input by 0x7fffffff (maximum signed int32) and writes the result. If the float input was slightly greater than 1, the integer wraps and becomes a very low value. Then in stage 3 we convert that s32pcm file to an ordinary s16pcm, but it's too late, the damage is done. You can verify that the damage happens at this stage easily by dropbox'ing a large file; watch /tmp and you'll see a directory like /tmp/rdaudioconvert/... appear in which the signed1 and signed2 files will be created. Once they stop getting larger but before they're deleted, copy them elsewhere for examination. I found that the signed1 file sounded fine but yielded complaints about clipped samples from sox (unsurprising; it's a float32 file so players finding values > 1 are clipping them), whilst the signed2 file exhibited unpleasant clipping due to the overflow issue; these mistakes were then replicated into the actual cut WAV file. The fix, then, is simple: just enable clipping by libsndfile when writing the stage 2 file. Any out-of-range floats will be clamped to the range of an int32 and everything will be fine. Alternatively, enable normalization; this happens befor e the float32 -> int32 conversion and so even with the mildest possible norm of -1dB it will ensure any marginally out-of-bounds floats are well in-bounds by the time they're written. The patch then is to insert: sf_command(dst_sf,SFC_SET_CLIPPING,NULL,SF_TRUE); in RDAudioConvert::Stage2Convert, after dst_sf is opened (similar to how SFC_NORM_FLOAT is set for src_sf). If like me you're looking at the rd-2.1.1 source tree that will be lib/rdaudioconvert.cpp:738 --Chris > Hi, > > An annoying bug today: I imported a file using a drop box, and found > that the result had some nasty crackling distortion in loud parts of > the recording. On further inspection, the original MP3 opened in > Audacity looks like this: > > http://cs448.user.srcf.net/clipbefore.png > > So we can see there's some clipping due to poor recording technique, > but the flat tops don't sound too bad on the recording, just > introduces a little buzz. However, the imported WAV file looks like this: > > http://cs448.user.srcf.net/clipafter.png > > So we can see that there has clearly been integer overflow turning > some of the very-nearly-max samples into very-nearly-min samples, and > turning the flat top into a very high frequency saw wave. This sounds > awful. > > So: the question is, where is this introduced. I reimported the file > using rdimport and it apparently inherited a normalization value from > the "ripper level" setting, default -13dBFS. This did not suffer the > problem, showing flat tops appropriately squashed. As the import web > service's workflow is to decode to WAV-signed-32, process, then encode > (in this case, just reduce depth to WAV-16), this suggests the problem > is introduced either during the processing stage or the encode. > > My suspicion was on the normalizer failing to notice an overflow, as > the dropbox is configured to use -1dBFS normalization. However I > suspect that's a red herring, as ps shows I am running > > rdimport --persistent-dropbox-id=12 --drop-box --log-mode > --normalization-level=0 --autotrim-level=0 --delete-source > --startdate-offset=0 --enddate-offset=0 Recordings > /music/playout-dropbox/recordings/* > > Which looks to me as though a different bug has disabled > normalization! Unless perhaps it ignores the norm-level=0 and gets > that setting from elsewhere? > > Anyway I will continue to investigate; has anyone seen this before? > Any hints on where to look next? The trickiest bit at the moment is > that the web services code is hard to harness and test... > > Chris _______________________________________________ Rivendell-dev mailing list [email protected] http://lists.rivendellaudio.org/mailman/listinfo/rivendell-dev
