psu wrote:
>
> Can someone provide some intuition as to why the gaussian distribution
> has the greatest entropy among all distributions of equal variance? (i'm
> specifically unclear as to why the gaussian would have greater entropy
> than say a uniform distribution of equal variance).
If you want a *proof*, I can recommend page 198 of David Williams'
excellent book "Weighing the Odds". ("Excellent" here means for the kind
of people who like proofs - it is NOT a book aimed at the "intuitive"
stats user, or even the intuitive stats user trying to develop rigor;
but it may be one of the best for the mathematician wanting to add
(theoretical) stats to his/her bag of tricks.) It is, alas, a
guess-and-check, deus-ex-machina proof that shows that
the more a distribution *isn't* Gaussian - by a less-than-transparent
measure - the less its entropy is.
If you want intuition, I can only suggest:
(a) that the uniform distribution *does* have maximum entropy for
distributions on a given interval; but if the domain is larger than the
interval of uniformity, the existence of a region where we're absolutely
sure
the variable is not going to be found makes the entropy "minus
infinity".
This is perfectly reasonable.
(b) On a half-open interval a constant cannot be a pdf. On the other
hand, the next simplest function is a linear function,and exp(a linear
function) can be a pdf. [The definition of entropy suffices to motivate
the exp.] Sure enough, the exponential distributions maximize entropy on
[0,infinity) (with mu specified).
(c) On the whole line, exp(ax+b) cannot be a pdf. Next in simplicity
come the quadratics, hence - wallah! Gaussians! which maximize entropy
with sigma and mu specified.
(d) I suspect that this handwave (and it is a big one) can be made
rigorous by means of the calculus of variations, turning the local
maximization of entropy subject to int[f(x)dx]=1 into a differential
equation to which these are the solutions. (But I've never tried.)
-Robert Dawson
.
.
=================================================================
Instructions for joining and leaving this list, remarks about the
problem of INAPPROPRIATE MESSAGES, and archives are available at:
. http://jse.stat.ncsu.edu/ .
=================================================================
