Re: [music-dsp] meassuring the difference
Your mean square error procedure is slightly incorrect. You should take the final signals from both processes, say A[1..n] and B[1..n], subtract them to get your error signal E[1..n], then the mean square error is the sum of the squared error over n. Sum( E[1..n]^2 ) / n This (MSE) is a statistical approach though and isn't necessarily a great way of measuring perceived acoustical differences. It depends on the nature of your signal but you may want to check the error in the frequency domain (weighted to specific frequency band if appropriate) rather than the time domain. -Original Message- From: music-dsp-boun...@music.columbia.edu [mailto:music-dsp-boun...@music.columbia.edu] On Behalf Of volker böhm Sent: 07 March 2013 15:10 To: A discussion list for music-related DSP Subject: [music-dsp] meassuring the difference dear all, i'm trying to meassure the difference between two equivalent but not identical processes. right now i'm feeding some test signals to both algorithms at the same time and subtract the output signals. now i'm looking for something to quantify the error signal. from statistics i know there is something like the mean squared error. so i'm squaring the error signal and take the (running) average. mostly i'm getting some numbers very close to zero and a gut feeling tells me i want to see those on a dB scale. so i'm taking the logarithm and multiply by 10, as i have already squared the values before. (as far as i can see, this is equivalent to a RMS meassurement). is there a correct/better/preferred way of doing this? next to a listening test, in the end i want to have a simple meassure of the difference of the two processes which is close to our perception of the difference. does that make sense? thanks for any comments, volker. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] meassuring the difference
On 3/7/13 10:10 AM, volker böhm wrote: dear all, i'm trying to meassure the difference between two equivalent but not identical processes. i sorta know know what you mean by this, maybe... but it would be interesting to see an articulated definition of what makes processes equivalent assuming we know what identical is. if equivalent is sounds the same ... right now i'm feeding some test signals to both algorithms at the same time and subtract the output signals. ... then i don't think this will work at all. a millisecond difference in delay will sound the same, but your difference signal will not be anywhere close to zero. there are all sorts of processes that are not simple filters, like reverbs, pitch-shifters, dynamics (AGC, compressor, limiter, gate), fuzz/distortion, that may sound equivalent but the outputs are quite different. now i'm looking for something to quantify the error signal. from statistics i know there is something like the mean squared error. so i'm squaring the error signal and take the (running) average. mostly i'm getting some numbers very close to zero wow! what are the equivalent but not identical processes? except for maybe differences in methods of rounding and quantization, or for simple filter, different structures or forms (like Direct Form 1 vs. State-Variable vs. Gold-Rader vs. Lattice/Ladder) i would not expect an error signal (i presume this is the difference of outputs) to be very close to zero. is this what your process is? i s'pose you could have a compressor with identical compression curve and where the delays are lined up very well and where the differences are in how compression levels are computed. even then, when the equivalent but not identical compressors pump, i would expect significant amplitude in the difference signal. i am curious what the equivalent but not identical processes are. and a gut feeling tells me i want to see those on a dB scale. so i'm taking the logarithm and multiply by 10, as i have already squared the values before. if it's the base-10 logarithm, that gets you dB. what makes 0 dB depends on how the signal is scaled before the log. (as far as i can see, this is equivalent to a RMS meassurement). how are you doing the M? is there a correct/better/preferred way of doing this? all depends on the alg. next to a listening test, in the end i want to have a simple meassure of the difference of the two processes which is close to our perception of the difference. does that make sense? it does, but a general analytic method that is close to our perception of the difference is a sorta holy grail for equivalent but not identical processes that are more sophisticated than just a filter or tapped delay or similar. -- r b-j r...@audioimagination.com Imagination is more important than knowledge. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] meassuring the difference
On 07.03.2013, at 16:27, Thomas Young wrote: Your mean square error procedure is slightly incorrect. You should take the final signals from both processes, say A[1..n] and B[1..n], subtract them to get your error signal E[1..n], then the mean square error is the sum of the squared error over n. Sum( E[1..n]^2 ) / n that's what i'm doing, no? This (MSE) is a statistical approach though and isn't necessarily a great way of measuring perceived acoustical differences. yes, this is what i'm suspecting. It depends on the nature of your signal but you may want to check the error in the frequency domain (weighted to specific frequency band if appropriate) rather than the time domain. thanks. will have to think a little bit about it. volker -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] meassuring the difference
On 07.03.2013, at 16:32, robert bristow-johnson wrote: now i'm looking for something to quantify the error signal. from statistics i know there is something like the mean squared error. so i'm squaring the error signal and take the (running) average. mostly i'm getting some numbers very close to zero wow! what are the equivalent but not identical processes? except for maybe differences in methods of rounding and quantization, or for simple filter, different structures or forms (like Direct Form 1 vs. State-Variable vs. Gold-Rader vs. Lattice/Ladder) i would not expect an error signal (i presume this is the difference of outputs) to be very close to zero. is this what your process is? i s'pose you could have a compressor with identical compression curve and where the delays are lined up very well and where the differences are in how compression levels are computed. even then, when the equivalent but not identical compressors pump, i would expect significant amplitude in the difference signal. i am curious what the equivalent but not identical processes are. fair enough. i see that my question was probably too vage in this respect. right now i'm doing simple stuff, yes, i'm comparing filter structures. i'm talking about differences that you hardly hear or might even not hear at all. and i'm interested if this i don't hear a difference maybe correleates to a measured error function. or put differently: how big can the error/difference be, without being perceivable (concerning a specific algorithm)? and a gut feeling tells me i want to see those on a dB scale. so i'm taking the logarithm and multiply by 10, as i have already squared the values before. if it's the base-10 logarithm, that gets you dB. what makes 0 dB depends on how the signal is scaled before the log. yes. (as far as i can see, this is equivalent to a RMS meassurement). how are you doing the M? i'm summing the samples and divide by the number. as my first attemps where in realtime, i was using a sliding window. is there a correct/better/preferred way of doing this? all depends on the alg. next to a listening test, in the end i want to have a simple meassure of the difference of the two processes which is close to our perception of the difference. does that make sense? it does, but a general analytic method that is close to our perception of the difference is a sorta holy grail for equivalent but not identical processes that are more sophisticated than just a filter or tapped delay or similar. yes, it's all about the holy grail! but i would be satisfied to find it for simple algos (right now). thanks for you comments. volker. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] meassuring the difference
On 3/7/13 1:41 PM, volker böhm wrote: On 07.03.2013, at 16:32, robert bristow-johnson wrote: now i'm looking for something to quantify the error signal. from statistics i know there is something like the mean squared error. so i'm squaring the error signal and take the (running) average. mostly i'm getting some numbers very close to zero wow! what are the equivalent but not identical processes? except for maybe differences in methods of rounding and quantization, or for simple filter, different structures or forms (like Direct Form 1 vs. State-Variable vs. Gold-Rader vs. Lattice/Ladder) i would not expect an error signal (i presume this is the difference of outputs) to be very close to zero. is this what your process is? i s'pose you could have a compressor with identical compression curve and where the delays are lined up very well and where the differences are in how compression levels are computed. even then, when the equivalent but not identical compressors pump, i would expect significant amplitude in the difference signal. i am curious what the equivalent but not identical processes are. fair enough. i see that my question was probably too vage in this respect. right now i'm doing simple stuff, yes, i'm comparing filter structures. i'm talking about differences that you hardly hear or might even not hear at all. with perfectly linear, time-invariant systems (LTI), the output signal is the theoretically-perfect output plus some error signal due to the internal quantization errors. so then you can line things up so that for two equivalent filters (both have the same H(z)), you can subtract one output from the other and you should get something that is very small and well-behaved. but this does not extend well to more sophisticated algs that are not LTI. and i'm interested if this i don't hear a difference maybe correleates to a measured error function. or put differently: how big can the error/difference be, without being perceivable (concerning a specific algorithm)? and a gut feeling tells me i want to see those on a dB scale. so i'm taking the logarithm and multiply by 10, as i have already squared the values before. if it's the base-10 logarithm, that gets you dB. what makes 0 dB depends on how the signal is scaled before the log. yes. (as far as i can see, this is equivalent to a RMS meassurement). how are you doing the M? i'm summing the samples and divide by the number. as my first attemps where in realtime, i was using a sliding window. that is, BTW, a simple moving average and it's a linear, time-invariant filter that happens to have a DC gain of 1 (or 0 dB). any LTI filter with DC gain of 1 will work. a simple 1-pole LPF will work for computing a sliding mean. is there a correct/better/preferred way of doing this? all depends on the alg. next to a listening test, in the end i want to have a simple meassure of the difference of the two processes which is close to our perception of the difference. does that make sense? it does, but a general analytic method that is close to our perception of the difference is a sorta holy grail for equivalent but not identical processes that are more sophisticated than just a filter or tapped delay or similar. yes, it's all about the holy grail! but i would be satisfied to find it for simple algos (right now). i think that someone in the AES has published ideas. dunno who exactly. i might check it out, but if it requires too much work, i might give up early. -- r b-j r...@audioimagination.com Imagination is more important than knowledge. -- dupswapdrop -- the music-dsp mailing list and website: subscription info, FAQ, source code archive, list archive, book reviews, dsp links http://music.columbia.edu/cmc/music-dsp http://music.columbia.edu/mailman/listinfo/music-dsp
