Re: [music-dsp] about entropy encoding
There's an entropy estimation equation that I occasionally see in papers about compression, for example in: "Binary-Valued Wavelet Decompositions of Binary Images" http://www.eurasip.org/Proceedings/Eusipco/1996/paper/mdsp_3.pdf The entropy estimation equation is this, found at the bottom of page 3: H(p) = -p*log2(p) - (1-p)*log2(1-p) (1) where "p is the number of nonzero pixels in the image, divided by the total number of pixels in the image" (the image being 1-bit black and white). This equation is the effectively the same as the one I gave in the earlier post, counting the occurrences of symbols, except it is applied to only two symbols: 0 and 1. It estimates the probability of a bit being set as 'p', which is the number of bits set divided by the total number of bits. In other words, it's effectively a counter - they just count the bits that are set. Then they normalize it by dividing by the total number of bits to get a value between 0-1, and call that "probability". If the probability of a bit being set is p, then the probability of a bit being zero is 1-p, as there are only two symbols, and the total probability is 1, so p + (1-p) = 1. So (1-p) in the equation refers to the probability of a bit being zero. Using Shannon's formula, the "entropy" of an event with probability "p" is -log2(p). So to estimate the total entropy of the bits in the image, they add the entropy of bits that are set, and the entropy of bit that are unset. So they multiply the entropy of ones -log2(p) by the probability of a bit being set (p) to get -p*log2(p), and the entropy of zeros -log2(1-p) multiplied by the probability of a bit being zero (1-p) to get -(1-p)*log2(1-p). If you sum these two, you get the above formula, and they call it 'bits per pixels'. I'm not saying this is necessarily correct, I'm just telling you how this formula is derived, and how the probabilities of each symbol are estimated. Apparently when these folks try to implement image compression, then they try to minimize this formula. When graphed, this function pretty much looks like an inverted parabola, peaking at p=0.5: http://morpheus.spectralhead.com/img/entropy-eq.png In addition, the entropy of bits is shown in red, and the entropy of zeros is shown in blue. So in other words, this graph shows the total entropy of a system with two symbols, where the probability of symbols are p and (1-p), respectively. When all bits are either zero or one, then the entropy is 0. So when doing image compression, they try to make all bits either one or zero. It also follows from the graph that uniform distribution of 1s and 0s would have maximum entropy. Let me add a comment here. In the paper linked above, these folks are trying to minimize the entropy of an image of a ship using a binary wavelet transformation, and using this formula, they find that the 'entropy' is decreased from 0.995 bpp (bit per pixel) to 0.212 bpp, using formula (1). My opinion is that they're probably wrong - if you see the two images, you'll see that the first one has very long continuous areas that are either 1 or 0. So, that image would be easily compressable using run-length encoding. In other words, the above entropy formula doesn't factor in the correlation between symbols (bits), namely that they're continuous areas of the same color (either 1 or 0), so it won't necessarily tell them, how well the image would compress using certain compression algorithms, like run-length encoding. In other words, formula (1) doesn't translate well to real-world results. If you remember from some time ago, I gave a formula that - instead of counting bits that are set, dividing it by the total number of bits, like equation (1) - rather, counts the number of bit _flips_, and divides it by the total amount of possible bitflips. So in that case, "p" would be: no. of bitflips p = --- total no. of bits - 1 So an entropy estimation formula that counts bitflips instead of bits, analogous to equation (1), could be: H(p) = -p*log2(p) - (1-p)*log2(1-p) (2) where p is the number of bitflips divided by the total number of bits minus one, in other words, it's the probability of two adjacent bits being flipped, and (1-p) is the probability of two adjacent bits being equal. So with this equation, when all nearby bits are flipped (pattern 010101010101...), it would give an entropy of zero. Equation (1) estimates entropy by counting the bits that are set, and equation (2) estimates entropy by counting bitflips. In my opinion, (2) would estimate better how well an image can be compressed, because that analyzes the correlation between adjacent symbols (bits), not just counts the number of bits set, which is a crude estimate. For example, if half of the bits are set, then equation (1) would give an entropy of 1, yet when the first half of the image is all white, and the second half is all black, then that doesn't
Re: [music-dsp] Sampling theorem extension
On 06-Jul-15 04:03, Sampo Syreeni wrote: On 2015-06-30, Vadim Zavalishin wrote: I would say the whole thread has been started mostly because of the exponential segments. How are they out of the picture? They are for *now* out, because I don't yet see how they could be bandlimited systematically within the BLEP framework. Didn't I describe this is my previous posts? But then evidently they can be bandlimited in all: just take a segment and bandlimit it. It's not going to be an easy math exercise, but it *is* going to be possible even within the distributional framework. I'd say even without one. A time-limited segment is in L2, isn't it? I don't think I'm good enough with integration to do that one myself. But you, Ethan and many others on this list probably are. Once you then have the analytic solution to that problem, I'm pretty sure you can tell from its manifest form whether the BLEP framework cut it. That would be a nice check, but I'm not sure I'd be able to derive an analytic closed-form expression for the related sum of the BLEPs which is what we need to compare against. But could you spot a mistake in my argument otherwise? Consider a piecewise-exponential signal being bandlimited by BLEP. That sort of implies an infinite sum of equal amplitude BLEPs, which probably can't converge. I think I have addressed exactly this convergence issue in my previous posts and in my paper. Furthermore, the convergence seems to be directly related to the bandlimitedness of the sine (see the paper). The same conditions hold for an exponential, hence my idea to define the "extended bandlimitedness" based on the BLEP convergence (or rather, the rolloff of the derivatives, which defines the BLEP convergence). Each of these exponentials can be represented as a sum of rectangular-windowed monomials (by windowing each term of the Taylor series separately). They can't: they are not finite sums, but infinite series, and I don't think we know how to handle such series right now. I meant "inifinite series of rectangular-windowed monomials". I'm not sure what specifically you are referring to by "we don't know how to handle them". We are just talking about pointwise convergence of this series. We can apply the BLEP method to bandlimit each of these monomials and then sum them up. We can handle each (actually sum of them) monomial. To finite order. But handling the whole series towards the exponential...not so much. Again, pointwise convergence is meant. If the sum converges then the obtained signal is bandlimited, right? If it does, yes. But I don't think it does. According to my paper, it does. Unless I did a mistake, the BLEP amplitudes roll off as 1/n, so if the derivatives (which are the BLEP gains) roll off exponentially decaying (which they do for a sine bandlimited to 1), the sum converges. Notice that this sufficient condition for the BLEP convergence is fulfilled if and only if the sine is bandlimited. I'm pretty sure you shouldn't be thinking about the bandlimited forms, now. The whole BLIT/BLEP theory hangs on the idea that you think about the continuous time, unlimited form first, and only then substitute -- in the very final step -- the corresponding bandlimited primitives. So it did, but what's wrong in doing the same for the monomial series? The sufficient convergence condition for the latter is that the derivatives of the exponent roll off sufficiently fast. But they don't, do they? Of course they do d^n exp(a*t) /dt^n = a^n * exp(a*t) so they roll off as a^n. The same for the sine. For a sine bandlimited to 1 we have a<1 and thus the BLEP sum converges. > When you snap an exponential back to zero, you necessarily snap back all of its derivatives. I'm not sure what you are referring to as "snapping". I mean, when you introduce a hard phase shift to a sine, you don't just modulate the waveform AM-wise. I do, if we consider the same in complex numbers. This is more or less what my paper is doing in the "ring modulation" approach. Especially when it gets bandlimited, the way you interpolate the waveform ain't gonna have just Diracs there, but Hilberts as well, and both of all orders... You lost me here a little bit. What's a Hilbert? 1/t? I thought it's the Fourier transform of a Heaviside. How is it a derivative of a Dirac? Furthermore, if we are talking in the spectral domain, we are going to have issues arising from the convergence of the infinite series in the time domain (you mentioned that the set of tempered distributions is not closed), that's why I specifically tried to stay in the time domain. That's the whole point: the bandlimitedness can be checked in the time domain, without even knowing the spectrum. Maybe the spectrum even doesn't exist for the full signal (like for an exponential), but we don't care, since our definition works only with time-limited segments of the signal. So to return to the
