[
https://issues.apache.org/jira/browse/MAHOUT-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288240#comment-13288240
]
Hudson commented on MAHOUT-986:
-------------------------------
Integrated in Mahout-Quality #1517 (See
[https://builds.apache.org/job/Mahout-Quality/1517/])
MAHOUT-986 Remove old LDA implementation from codebase (Revision 1345736)
Result = FAILURE
ssc : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1345736
Files :
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDADocumentTopicMapper.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDADriver.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDAInference.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDAReducer.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDASampler.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDAState.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDAUtil.java
*
/mahout/trunk/core/src/main/java/org/apache/mahout/clustering/lda/LDAWordTopicMapper.java
*
/mahout/trunk/core/src/test/java/org/apache/mahout/clustering/ClusteringTestUtils.java
*
/mahout/trunk/core/src/test/java/org/apache/mahout/clustering/lda/TestLDAInference.java
*
/mahout/trunk/core/src/test/java/org/apache/mahout/clustering/lda/TestMapReduce.java
* /mahout/trunk/src/conf/driver.classes.props
* /mahout/trunk/src/conf/lda.props
* /mahout/trunk/src/conf/ldatopics.props
> OutOfMemoryError in LanczosState by way of SpectralKMeans
> ---------------------------------------------------------
>
> Key: MAHOUT-986
> URL: https://issues.apache.org/jira/browse/MAHOUT-986
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.6
> Environment: Ubuntu 11.10 (64-bit)
> Reporter: Shannon Quinn
> Assignee: Shannon Quinn
> Priority: Minor
> Fix For: 0.7
>
>
> Dan Brickley and I have been testing SpectralKMeans with a dbpedia dataset (
> http://danbri.org/2012/spectral/dbpedia/ ); effectively, a graph with
> 4,192,499 nodes. Not surprisingly, the LanczosSolver throws an
> OutOfMemoryError when it attempts to instantiate a DenseMatrix of dimensions
> 4192499-by-4192499 (~17.5 trillion double-precision floating point values).
> Here's the full stack trace:
> {quote}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at org.apache.mahout.math.DenseMatrix.<init>(DenseMatrix.java:50)
> at
> org.apache.mahout.math.decomposer.lanczos.LanczosState.<init>(LanczosState.java:45)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.run(SpectralKMeansDriver.java:146)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.run(SpectralKMeansDriver.java:86)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.main(SpectralKMeansDriver.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> {quote}
> Obviously SKM needs a more sustainable and memory-efficient way of performing
> an eigen-decomposition of the graph laplacian. For those who are more
> knowledgeable in the linear systems solvers of Mahout than I, can the Lanczos
> parameters be tweaked to negate the requirement of a full DenseMatrix? Or
> should SKM move to SSVD instead?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
