I believe the appearance of a paradox lies in alternatate uses of the words "information" or "informative".
Shannon was right that a white-noise-like signal contains the most information content. If the signal were truly generated from a random process, then this may seem odd, but the observer of the signal cannot know if it is a truly random string, or just an optimally compressed message. In fact, an optimally compressed message is indistinguishable from a truly random string (otherwise it could be compressed further). If one knows the underlying process generating the signal, one might try to characterize the "structure" or "complexity" or "randomness" of the signal -- there are various ways to define all of these charactertistics. This is an approach taken in the complex systems literature (see, for example, http://www.santafe.edu/sfi/publications/wpabstract/199707064), and I believe also in CS theory (see, for example, L. Antunes, L. Fortnow, and D. van Melkebeek. Computational depth. In Proceedings of the 16th IEEE Conference on Computational Complexity, pages 266-273. IEEE, New York, 2001. http://external.nj.nec.com/homepages/fortnow/papers/depth.ps). Usually they strive to define complexity to be minimum both for simple patterns and for purely random strings, with maximum complexity falling somewhere in between, which is more along the lines of your first intuition of "informative signals" as "one which contains patterns". Hope this helps, Dave On Mon, 28 Oct 2002, Rizwan Choudrey wrote: > Dear all, > > I wondered if anyone could help with a paradox at the heart of my > understanding of entropy, information and pattern recognition. > > I understand an informative signals as one which contains patterns, as > opposed to radomly distributed numbers e.g. noise. Therefore, I > equate information with structure in the signals distribution. However, > Shannon equates information with entropy, which is maximimum when each > symbol in the signal is equally as likely as the next i.e. a distribution > with no `structure'. These views are contradictory. > > What am I misisng in my understanding? > > Many thanks in advance, > Riz > > > Rizwan Choudrey > Robotics Group > Department of Engineering Science > University of Oxford > 07956 455380 >
