I recently read Chapter 6 in Good's book "Probability and The Weighing of
Evidence."
In Section 6.9, he says that he just learned about Shannon's paper (it
came out as he was publishing his book) and he summarizes the idea
behind entropy/information, in a very concise and intuitive way:
We want a function I, which measures the amount of information we
obtain when we learn an event E whose probability is p. The function
is expected to satisfy two conditions:
1. It must be decreasing in the probability p (unlikely things
tells us more than likely things).
2. The amount of information provided by two independent events
should be the sum of the separate amounts.
The only functions satisfying these conditions are of the form - log p.
Now consider an experiment whose possible outcomes are mutually
exclusive and exhaustive events with probabilities p1..pn. The
expected amount of information from the experiment is then:
- (p1 log p1 + .... + pn log n).
This is called by Shannon the entropy of an experiment.
Hope you will find this useful.
Adnan
