Re: [flac-dev] 64-bit residuals
On Tue, Apr 05, 2022 at 11:41:12AM +0200, Martijn van Beurden wrote: > Extrapolating from these results, it seems unlikely that increasing > the residual limit beyond a 32-bit int would bring significant > compression improvement for music sources. Interesting. I wasn't expecting that. I tried your code with some 16-bit audio expanded with 16 bits of noise and I don't see a large difference here either. Thanks for you work on this. -- Miroslav Lichvar ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op vr 1 apr. 2022 om 09:19 schreef Martijn van Beurden : > I'll start working on a patch to include stereo decorrelation and > handling fixed subframes for 32-bit FLAC in both decoder and encoder > to do some more testing. It seems that those two changes are worth the > extra encoding and decoding complexity, while adding handling for > > 32-bit residuals (which need new bitreader and bitwriter functions) > does not. A still rough (but fully functional) patch to libFLAC to enable 32-bit decoding and encoding is here: https://github.com/ktmf01/flac/tree/32bit-encoder-decoder This implements encoding and decoding to and from all 4 subframe types, with stereo decorrelation (i.e. 33 bps subframes). The decoder rejects residuals exceeding 32-bit int (still work in progress) and the encoder takes care it doesn't produce residuals exceeding 32-bit int. I compared encodes with and without stereo decorrelation, and with even lower residual limits. Comparison was done with the same corpus as mentioned earlier in this thread. Results are as follows: With stereo decorrelation, residual limited to 32-bit int: 1161MB (0.01% verbatim) Without stereo decorrelation, residual limited to 32-bit int: 1180MB (0.04% verbatim) With stereo decorrelation, residual limited to "31-bit int": 1162MB (0.6% verbatim) Without stereo decorrelation, residual limited to "31-bit int": 1183MB (1.7% verbatim) With stereo decorrelation, residual limited to "30-bit int": 1172MB (6.9% verbatim) Without stereo decorrelation, residual limited to "30-bit int": 1197MB (10% verbatim) Extrapolating from these results, it seems unlikely that increasing the residual limit beyond a 32-bit int would bring significant compression improvement for music sources. Enabling stereo decorrelation does bring quite some benefit, but at the cost of increased encoder and decoder complexity (see patch for details). Kind regards, Martijn van Beurden ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op wo 30 mrt. 2022 om 16:02 schreef Martijn van Beurden : > One added advantage of setting both limits for 32-bit encodings (not > allowing stereo decorrelation and limiting residuals to 32-bit) is > that libFLAC from 1.2.1 onwards and ffmpeg since May 2015 are > "forward-compatible". ffmpeg actually needs limiting the residual to > 31-bits for that to work. A working patch for the ffmpeg FLAC encoder > can be found here: > https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220108142437.756529-1-mva...@gmail.com/ > In fact, the files generated with this patch play on a few hardware > devices I tested them with that use a libFLAC-derived decoder. I just realised I forgot to mention one additional constraint this patch imposes for backwards-compatibility: those files also cannot include fixed predictors. So there are four constraints for backward-compatible 32-bit streams: no fixed subframes, no stereo decorrelation, no predictions exceeding a 32-bit int and no residuals exceeding a "31-bit int". Summing this all up, this seems to be getting out of hand just for the sake of being compatible with software that was never meant to decode these streams in the first place I'll start working on a patch to include stereo decorrelation and handling fixed subframes for 32-bit FLAC in both decoder and encoder to do some more testing. It seems that those two changes are worth the extra encoding and decoding complexity, while adding handling for > 32-bit residuals (which need new bitreader and bitwriter functions) does not. Kind regards, Martijn van Beurden ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op do 31 mrt. 2022 om 09:23 schreef Miroslav Lichvar : > Can you post some examples of the impact on compression ratio? Do the > samples actually use all of the 32 bits? I have no experince with > this. I did some tests with ffmpeg with the patch I linked earlier. This patch limits the residuals to "31-bit ints" and the LPC predictions to 32-bit ints. I varied the residual limit between "30-bit int", "31-bit int" and 32-bit int. Limiting to "31-bit int" is needed for backward compatibility with ffmpeg. I compressed a test corpus with this patched ffmpeg, and measured file size of the compressed corpus and the total number of verbatim frames, which indicates how often the encoder uses verbatim frames as fall-back in case it cannot find a predictor with residuals that fall within the stated limits. Test corpus was created from 16-bit files upscaled by SoX with the hilbert filter to fill the added 16-bit. Corpus consists of 20 tracks of mostly music encompassing a wide range of genres, from solo piano through heavy metal, and also including two non-music recordings. set of 32-bit WAV: 1749MB 32-bit FLAC with 32-bit residuals, no stereo decorrelation: 1215MB (2.4% verbatim) 32-bit FLAC with 31-bit residuals, no stereo decorrelation: 1219MB (4.5% verbatim) 32-bit FLAC with 30-bit residuals, no stereo decorrelation: 1234MB (27% verbatim) To assess the influence of limiting the LPC predictions to 32-bit ints and limiting stereo decorrelation, I also tested with 31 bit per sample files. 31-bit FLAC with 32-bit residuals, no stereo correlation: 1160MB (0.0% verbatim) 31-bit FLAC with 32-bit residuals: 1141MB (0.0% verbatim) 31-bit FLAC with 31-bit residuals, no stereo decorrelation: 1161MB (0.6% verbatim) 31-bit FLAC with 31-bit residuals: 1141MB (0.3% verbatim) 31-bit FLAC with 30-bit residuals, no stereo decorrelation: 1165MB (6.2% verbatim) 31-bit FLAC with 30-bit residuals: 1144MB (3.7% verbatim) >From this data I think I can conclude the following: - Limiting residuals to "31-bit ints" hampers compression by 0.25%. >From the results with 31-bit FLAC files it seems capping the residual at 32-bit ints does negligible harm to compression, but capping them to 31-bit does harm it a bit. So, 0.25%-point is calculated as (1219-1215)/1749 - Limiting predictions to 32-bit ints hampers compression by another 0.25%-point. Comparing the results of 31-bit FLAC with 32-bit FLAC and combining that with the previous conclusion, it seems the 2.4% verbatim frames in 32-bit FLAC with 32-bit residuals seem to come mostly from this limit. - Limiting stereo decorrelation seems to hamper compression by 1.1%-point. This is calculated from the difference between the 31-bit FLACs with and without stereo decorrelation. So, from this limited performance testing, it seems imposing these limits causes compression to fall by 1.6%-point. It is a little more than I expected, but performance is still quite reasonable. Kind regards, Martijn van Beurden ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
On Wed, Mar 30, 2022 at 04:02:11PM +0200, Martijn van Beurden wrote: > Granted, it won't provide the best possible compression, but 32-bit > PCM is itself already an extremely inefficient format. So I think the > choice here is between slightly better compression and a simpler > encoder versus backward compatibility with a lot of decoders and no > added decoder complexity. Can you post some examples of the impact on compression ratio? Do the samples actually use all of the 32 bits? I have no experince with this. If compression of 16-bit audio was limited to 16-bit residuals, I suspect that would have a significant impact, so I'm wondering how it is different with 32-bit audio. -- Miroslav Lichvar ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op wo 30 mrt. 2022 om 11:03 schreef Miroslav Lichvar : > I think that should apply only to <=24 bits. If I understand it > correctly, with 32 bits it would be a real limitation, not hit only > with specifically crafted encodings. > > [...] > > No, that doesn't seem acceptable to me. > One added advantage of setting both limits for 32-bit encodings (not allowing stereo decorrelation and limiting residuals to 32-bit) is that libFLAC from 1.2.1 onwards and ffmpeg since May 2015 are "forward-compatible". ffmpeg actually needs limiting the residual to 31-bits for that to work. A working patch for the ffmpeg FLAC encoder can be found here: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220108142437.756529-1-mva...@gmail.com/ In fact, the files generated with this patch play on a few hardware devices I tested them with that use a libFLAC-derived decoder. Granted, it won't provide the best possible compression, but 32-bit PCM is itself already an extremely inefficient format. So I think the choice here is between slightly better compression and a simpler encoder versus backward compatibility with a lot of decoders and no added decoder complexity. > The intended status of the current FLAC draft is informational, so > there doesn't have to be any existing implementation that supports all > bit depths, right? Actually, the intended status was indeed information in the last few drafts, but this will be changed back to standards track, as it was in the first few drafts. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
On Tue, Mar 29, 2022 at 04:57:10PM +0200, Martijn van Beurden wrote: > Op di 29 mrt. 2022 om 11:43 schreef Miroslav Lichvar : > > The third option makes most sense to me. I don't think we should > > complicate decoders by requiring them to support 64-bit residuals only > > because it's technically possible to encode such a stream. > > Would you argue this limitation is imposed on all possible FLAC > streams, or just for PCM inputs with a bit depth up to and including > 24? Or should this also apply to 32-bit streams? I think that should apply only to <=24 bits. If I understand it correctly, with 32 bits it would be a real limitation, not hit only with specifically crafted encodings. > A similar problem in the spec represents itself with 32-bit PCM > inputs. When applying stereo decorrelation, a transformation to side a > subframe bps of 33, which is very inconvenient. Should stereo > decorrelation be forbidden for 32 bps inputs? No, that doesn't seem acceptable to me. > While these distinctions might seem unimportant and 32-bit streams > folly, there is currently an effort underway to make FLAC and IETF RFC > standard. I think that the decoder of the reference implementation > (libFLAC) should support all features the format has before the > standard becomes final. As the FLAC format has always supported 32 bps > streams but no encoder for these exists, I think this requires some > extra care. The intended status of the current FLAC draft is informational, so there doesn't have to be any existing implementation that supports all bit depths, right? -- Miroslav Lichvar ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op di 29 mrt. 2022 om 11:43 schreef Miroslav Lichvar : > The third option makes most sense to me. I don't think we should > complicate decoders by requiring them to support 64-bit residuals only > because it's technically possible to encode such a stream. Would you argue this limitation is imposed on all possible FLAC streams, or just for PCM inputs with a bit depth up to and including 24? Or should this also apply to 32-bit streams? A similar problem in the spec represents itself with 32-bit PCM inputs. When applying stereo decorrelation, a transformation to side a subframe bps of 33, which is very inconvenient. Should stereo decorrelation be forbidden for 32 bps inputs? While these distinctions might seem unimportant and 32-bit streams folly, there is currently an effort underway to make FLAC and IETF RFC standard. I think that the decoder of the reference implementation (libFLAC) should support all features the format has before the standard becomes final. As the FLAC format has always supported 32 bps streams but no encoder for these exists, I think this requires some extra care. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
On Thu, Mar 24, 2022 at 05:05:02PM +0100, Martijn van Beurden wrote: > 3) Add a note to the FLAC spec that residuals should not exceed > 32-bit, and add a mechanism to the encoder to ascertain this. In case > the encoder finds a residual exceeding the 32-bit range, it defaults > to using a verbatim subframe. > > Any thoughts? The third option makes most sense to me. I don't think we should complicate decoders by requiring them to support 64-bit residuals only because it's technically possible to encode such a stream. Decoders can produce wrong output or reject the stream as invalid, but they should not crash. -- Miroslav Lichvar ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Op do 24 mrt. 2022 om 18:01 schreef Brian Willoughby : > > Can you briefly explain the consequences of this unexpected residual? > Of course. The WAV file I've used now does not cause problems when compressed with any release version of libFLAC as far as I can tell. For now, this is more of a decoder-side problem: encoders other than libFLAC or a future libFLAC might create such residuals, as it is not limited in the spec. The problem files in the github issue have been created with an experimental version of libFLAC with some compression improvements I've been working on the past two years. This experimental encoder implements a so called iteratively reweighted least squares solver (IRLS) to find a least absolute deviation (LAD) solution to the problem of finding an optimal predictor, whereas current libFLAC uses an approach that more or less resembles a least squares solution. When using a LAD solution, the average residual is smaller at the cost of the maximal residual being much larger. Therefore I do not expect this to happen with released versions of libFLAC, not even with crafted material. The _ultrawide functions would be used when the formula subframe_bps + qlp_coeff_precision + FLAC__bitmath_ilog2(order) - quantization_level returns a value larger than 32. This will never happen on 16-bit files, not often on 24-bit files and quite often on 32-bit files, if encoding and decoding 32-bit files is ever implemented. If at some point this experimental branch is merged into a release version, the test suite will need to be expanded to cover this. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Can you briefly explain the consequences of this unexpected residual? Will a 24-bit crafted file be compressed in a lossy way? Are there tests in the FLAC suite that would reveal this? Can we add such a crafted 24-bit file to the test suite? When would the new _ultrawide variants of existing functions be used? ... I ask because my use of FLAC is entirely 24-bit, so I appreciate the discovery of this potential issue. Brian Willoughby On Mar 24, 2022, at 09:05, Martijn van Beurden wrote: > Hi all, > > I just filed an issue on github: > https://github.com/xiph/flac/issues/306 Long story short: libFLAC > currently uses 32-bit signed integers for residuals, but using certain > 24-bit (crafted) material can overflow this. For examples and an > explanation see the github issue. > > I'm sending this e-mail because I'd like to attract some extra > attention towards it. I'm working on patching this, but the residual > is actually exported through the API (as part of the subframe) so I'd > like to have some discussion on what would be the best way to fix > this. I see the following options: > > 1) Change all residual handling from 32-bit int to 64-bit int. This > might incur a performance penalty and it might invalidate certain > approaches now used with intrinsics. It also presents an API change > 2) Change the current 32-bit signed integer arrays which are used for > residual handling to a union within a struct. This union would hold > both a 32-bit int and a 64-bit int variant of the residual, and the > struct would include a bool to clarify which of the two is being used. > For each function manipulating residuals (bitreader, bitwriter, lpc > restore etc.) a new _ultrawide variant is written and at several > places in the code a decision has to be added whether to use the > _ultrawide variant or the existing ones. This changes the API a little > more, but makes it able to keep the current intrinsics accelerated > functions > 3) Add a note to the FLAC spec that residuals should not exceed > 32-bit, and add a mechanism to the encoder to ascertain this. In case > the encoder finds a residual exceeding the 32-bit range, it defaults > to using a verbatim subframe. > > Any thoughts? ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 64-bit residuals
Hi Martijn, I just want to thank you for your thoughtful attention to these details. Much appreciated! Jim On Thu, March 24, 2022 9:05 am, Martijn van Beurden wrote: > Hi all, > > > I just filed an issue on github: > https://github.com/xiph/flac/issues/306 Long story short: libFLAC > currently uses 32-bit signed integers for residuals, but using certain > 24-bit (crafted) material can overflow this. For examples and an > explanation see the github issue. > > I'm sending this e-mail because I'd like to attract some extra > attention towards it. I'm working on patching this, but the residual is > actually exported through the API (as part of the subframe) so I'd like to > have some discussion on what would be the best way to fix this. I see the > following options: > > 1) Change all residual handling from 32-bit int to 64-bit int. This > might incur a performance penalty and it might invalidate certain > approaches now used with intrinsics. It also presents an API change 2) > Change the current 32-bit signed integer arrays which are used for > residual handling to a union within a struct. This union would hold both a > 32-bit int and a 64-bit int variant of the residual, and the > struct would include a bool to clarify which of the two is being used. For > each function manipulating residuals (bitreader, bitwriter, lpc restore > etc.) a new _ultrawide variant is written and at several places in the > code a decision has to be added whether to use the _ultrawide variant or > the existing ones. This changes the API a little more, but makes it able > to keep the current intrinsics accelerated functions 3) Add a note to the > FLAC spec that residuals should not exceed > 32-bit, and add a mechanism to the encoder to ascertain this. In case > the encoder finds a residual exceeding the 32-bit range, it defaults to > using a verbatim subframe. > > Any thoughts? > ___ > flac-dev mailing list flac-dev@xiph.org > http://lists.xiph.org/mailman/listinfo/flac-dev ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
