[
https://issues.apache.org/jira/browse/MAHOUT-121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740646#action_12740646
]
Nicolás Fantone commented on MAHOUT-121:
----------------------------------------
Jira is hardly the place to discuss this (almost moral) issues, as this tends
to be a matter of personal choice most of the time. But I feel the urge to
reply: some of you have really helped me out here at work with my project.
{quote}
Are we talking about the same patch? sections like this?
{quote}
Yes we are. And while we are taking a look at that quoted piece of code, I
shall mention that the declaration of distance and clusterCenter outside the
loop isn't the only thing being changed in the patch. There's also an
elimination of a an unnecessary boolean evaluation in the if statement.
{quote}
I think we're not talking about the same thing, since you pick up on the point
of creating Strings in a loop - which both my examples did.
{quote}
We may not. Perhaps, my String example wasn't the happiest of them all - but it
certainly IS showing how costly it is in JAVA to just instantiate and
initialize an object. Furthermore, I thought I gave a counterargument to what
you were implying with those examples.
{quote}
I am further making the point that there is not a tradeoff between readability
and speed here - putting the declaration outside the loop actually makes both
worse.
{quote}
I can tell none of my propositions were convincing enough. Maybe someone
else's?
http://weblogs.java.net/blog/ddevore/archive/2006/08/declare_variabl_1.html
> Speed up distance calculations for sparse vectors
> -------------------------------------------------
>
> Key: MAHOUT-121
> URL: https://issues.apache.org/jira/browse/MAHOUT-121
> Project: Mahout
> Issue Type: Improvement
> Components: Matrix
> Reporter: Shashikant Kore
> Assignee: Grant Ingersoll
> Attachments: Canopy_Wiki_1000-2009-06-24.snapshot, doc-vector-4k,
> MAHOUT-121-cluster-distance.patch, MAHOUT-121-distance-optimization.patch,
> MAHOUT-121-new-distance-optimization.patch, mahout-121.patch,
> MAHOUT-121.patch, MAHOUT-121.patch, MAHOUT-121.patch, MAHOUT-121.patch,
> MAHOUT-121.patch, mahout-121.patch, MAHOUT-121jfe.patch, Mahout1211.patch
>
>
> From my mail to the Mahout mailing list.
> I am working on clustering a dataset which has thousands of sparse vectors.
> The complete dataset has few tens of thousands of feature items but each
> vector has only couple of hundred feature items. For this, there is an
> optimization in distance calculation, a link to which I found the archives of
> Mahout mailing list.
> http://lingpipe-blog.com/2009/03/12/speeding-up-k-means-clustering-algebra-sparse-vectors/
> I tried out this optimization. The test setup had 2000 document vectors
> with few hundred items. I ran canopy generation with Euclidean distance and
> t1, t2 values as 250 and 200.
>
> Current Canopy Generation: 28 min 15 sec.
> Canopy Generation with distance optimization: 1 min 38 sec.
> I know by experience that using Integer, Double objects instead of primitives
> is computationally expensive. I changed the sparse vector implementation to
> used primitive collections by Trove [
> http://trove4j.sourceforge.net/ ].
> Distance optimization with Trove: 59 sec
> Current canopy generation with Trove: 21 min 55 sec
> To sum, these two optimizations reduced cluster generation time by a 97%.
> Currently, I have made the changes for Euclidean Distance, Canopy and KMeans.
>
> Licensing of Trove seems to be an issue which needs to be addressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
