On Wed, 15 Dec 2004, Wiener, Matthew wrote: >It sounds like "clara" in package cluster might help.
Cheers, this looks just the ticket. How should I choose k though? Dan. > >Regards, > >Matt Wiener > >-----Original Message----- >From: [EMAIL PROTECTED] >[mailto:[EMAIL PROTECTED] On Behalf Of Dan Bolser >Sent: Wednesday, December 15, 2004 6:37 AM >To: R mailing list >Subject: [R] Massive clustering job? > > > >Hi, > >I have ~40,000 rows in a database, each of which contains an id column and >20 additional columns of count data. > >I want to cluster the rows based on these count vectors. > >Their are ~1.6 billion possible 'distances' between pairs of vectors >(cells in my distance matrix), so I need to do something smart. > >Can R somehow handle this? > >My first thought was to index the database with something that makes >nearest neighbour lookup more efficient, and then use single linkage >clustering. Is this kind of index implemented in R (by default when using >single linkage)? > >Also 'grouping' identical vectors is very easy. I tried making groups more >fuzzy by using a hashing function over the count vectors, but my hash was >too crude. Any way to do fuzzy grouping in R which scales well? > >For example, removing identical vectors gives me ~30,000 rows (and ~900 >million pairs of distances). As an example of how fast I can group, the >above query took 0.13 seconds in mysql (using an index over every element >in the vector). However, if I tried to calculate a distance between every >pair of non identical vectors (lets say I can calculate ~1000 eutlidian >distances per second) it would take me ~10 days just to calculate the >distance matrix. > >Sorry for all the information. Any suggestions on how to cluster such a >huge dataset (using R) would be appreciated. > >Cheers, >Dan. > >______________________________________________ >[EMAIL PROTECTED] mailing list >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide! >http://www.R-project.org/posting-guide.html > > > >------------------------------------------------------------------------------ >Notice: This e-mail message, together with any attachments, contains >information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New >Jersey, USA 08889), and/or its affiliates (which may be known outside the >United States as Merck Frosst, Merck Sharp & Dohme or MSD and in Japan, as >Banyu) that may be confidential, proprietary copyrighted and/or legally >privileged. It is intended solely for the use of the individual or entity >named on this message. If you are not the intended recipient, and have >received this message in error, please notify us immediately by reply e-mail >and then delete it from your system. >------------------------------------------------------------------------------ > ______________________________________________ [EMAIL PROTECTED] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
