A very wise comment about clustering nlp data from someone who knows a lot about it (Fernando Pereira of UPenn - a quick google search will show off his qualifications if you aren't familiar with him). This is in reply to a message about Weka's implementation of clustering, but it certainly applies in general. Note that Fernando stresses the need for dimensionality reduction, which is exactly what we're trying to do in SenseClusters with SVD.
-- Ted Pedersen http://www.d.umn.edu/~tpederse ---------- Forwarded message ---------- Date: Sun, 27 Jun 2004 16:43:43 -0400 From: Fernando Pereira <[EMAIL PROTECTED]> To: Jonathan Elsas <[EMAIL PROTECTED]> Cc: [EMAIL PROTECTED] Subject: Re: [Wekalist] EM cluster empty exception On Jun 23, 2004, at 4:07 PM, Jonathan Elsas wrote: > I've received this exception a lot with the EM implementation in weka. > I had previously posted an email to the weka-list re. this problem > (although I can't seem to find it in the wekalist archives online), > but no one responded. We've been using the EM for text data, which is > also very high-dimensional (and sparse). I don't know if that has > something to do with the EM implementation producing this exception. Clustering text data or any very high dimension sparse data is very tricky, and it's unlikely that a generic latent variable clustering model will do anything useful without careful data preparation. There are (at least) two reasons for this. First, the predominance of zeros in the data vectors causes drastic overfitting, which may be the cause of the empy cluster exception (I don't know the details of the WEKA method, but I've seen similar problems with other implementations). Second, a standard Gaussian mixture model is hard to fit in high dimension <http://charlotte.ucsd.edu/users/dasgupta/papers/focs2.ps>. Some form of dimensionality reduction is critical for getting useful results <http://charlotte.ucsd.edu/users/dasgupta/papers/random.ps>. -- F _______________________________________________ Wekalist mailing list [EMAIL PROTECTED] https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist ------------------------ Yahoo! Groups Sponsor --------------------~--> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar. Now with Pop-Up Blocker. Get it for free! http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/x3XolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
