[
https://issues.apache.org/jira/browse/MAHOUT-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660478#comment-13660478
]
Ted Dunning commented on MAHOUT-1218:
-------------------------------------
Each mapper has to do a good sketch of its own data.
This implies that -km should have a minimum value of k log(n/m). This gives a
total number of points to deal with of k m log (n/m).
But argmax_m k m log(k/m) = k/e so the maximum number of points is n k / e
which is bigger than n for most interesting values of k. Even for a very small
range of m, say m <= 100, we have problems.
My guess is that we will need to have a fancier reducer that recursively
reduces the centroids before clustering. Or combiners. Or somehting.
> Streamimg k-means fails when the number of clusters specified is <= estimated
> map clusters
> ------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1218
> URL: https://issues.apache.org/jira/browse/MAHOUT-1218
> Project: Mahout
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 0.8
> Reporter: Suneel Marthi
> Assignee: Suneel Marthi
> Fix For: 0.8
>
>
> Running Streaming k-means with CosineDistanceMeasure, Fast Projection Search,
> number of clusters k= 60, number of estimated map clsuters -km = 60.
> {Code}
> Exception in thread "main" java.lang.IllegalArgumentException: Invalid number
> of estimated map clusters; There must be more than the final number of
> clusters (k log n vs k)
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> at
> org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansDriver.configureOptionsForWorkers(StreamingKMeansDriver.java:327)
> at
> org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansDriver.configureOptionsForWorkers(StreamingKMeansDriver.java:280)
> at
> org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansDriver.run(StreamingKMeansDriver.java:227)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansDriver.main(StreamingKMeansDriver.java:472)
> {Code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
