Ted Dunning wrote:
This is an interesting point that actually came up in my data mining talk at Yahoo! a couple of weeks back. At the time, I did not completely get the impact of his question, but now I do. One of their senior statisticians (the only other person in the room with any gray hair<grin>), posed this conundrum for Mahout clustering:The slides are pretty clear. The titles of other talks there lead to a good batch of articles to read.The basic idea is that sampling rows and/or columns can give approximately correct matrix products very quickly. This same technique can be used to give SVD like results quickly.
1. Beyond 20 or so dimensions in the dataset, the "curse of dimensionality" limits what can be reasonably gained from clustering because the data points become so far apart in n-space. This limits the value of wide dataset handling in Mahout. 2. Since very large datasets can be effectively clustered using sampling techniques (as in the above) to determine the clusters, the benefit of using all the points for deep dataset cluster determination is also limited.
I still do not know how to respond to it, but offer it to the list for discussion. Of course, once the cluster centers are known, Mahout offers a faster way to actually cluster all of the points, if this is needed for the application.
Jeff
begin:vcard fn:Jeff Eastman n:Eastman;Jeff org:Windward Solutions Inc. adr:;;;Los Altos;CA;;USA email;internet:[EMAIL PROTECTED] title:President tel;pager:http://[EMAIL PROTECTED] tel;cell:+1.415.298.0023 x-mozilla-html:TRUE url:http://www.windwardsolutions.com version:2.1 end:vcard
PGP.sig
Description: PGP signature
