> > Maximum entropy principle is about the idea that given a number of > > equivalent prospective models, you should pick the one with the > > highest entropy. > I guess EM also tries to do the same, ie tries to maximize the > likelihood of incomplete pdf match the complete pdf iteratively for > said parameter.
You misunderstand maximum entropy. Assume the true pdf p(x), and the modelled pdf q(x). The purpose of cross-entropy minimization is to find a q(x) so that the KL-divergence between the two pdfs will be minimal: q := argmin_q' D(p || q') Assume that q is a parametric model, with parameter t. Then q(x)=p(x | t). The purpose of maximum likelihood (ML) is to pick the model (or parameters) given which the data are likeliest: t := argmax_t' p(x | t') q(x) := p(x | t) Although these two look similar, they are *not* the same! Let's now imagine that we have a number of models, q1(x), q2(x), q3(x). The purpose of MaxEnt is to pick the one with highest entropy H(q): q := argmax_q' H(q) H(q) is the entropy of a particular pdf q: H(q) := -Integrate[q(x)log q(x)]dx So there is very little similarity between MaxEnt and ML. Again, EM is only a particular approach to ML. There are some connections, but they're a tad more intricate (Csiszar & Tusnady 1984). -- mag. Aleks Jakulin http://ai.fri.uni-lj.si/aleks/ Artificial Intelligence Laboratory, Faculty of Computer and Information Science, University of Ljubljana. . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
