[
https://issues.apache.org/jira/browse/MAHOUT-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288595#comment-13288595
]
Dan Brickley commented on MAHOUT-986:
-------------------------------------
Thanks for the fix. Testing it, I rediscover a problem from offlist thread w/
Shannon.
On my setup (pseudo-cluster on one oxs laptop, right now), if there is an _ /
underscore in the input/ filename, I get this error:
2/06/04 07:21:52 INFO common.VectorCache: Loading vector from:
spectral/output/calculations/diagonal/part-r-00000
Exception in thread "main" java.util.NoSuchElementException
at
com.google.common.collect.AbstractIterator.next(AbstractIterator.java:152)
at
org.apache.mahout.clustering.spectral.common.VectorCache.load(VectorCache.java:115)
2/06/04 07:21:52 INFO common.VectorCache: Loading vector from:
spectral/output/calculations/diagonal/part-r-00000
Exception in thread "main" java.util.NoSuchElementException
at
com.google.common.collect.AbstractIterator.next(AbstractIterator.java:152)
at
org.apache.mahout.clustering.spectral.common.VectorCache.load(VectorCache.java:115)
This (seems to, I'm still waiting to confirm but the run got further than
before) works for a fix to that; however as nobody else (including Shannon)
seems to have hit this, I didn't raise a JIRA yet.
grunt> cd spectral
grunt> cd input
grunt> ls
hdfs://localhost:9000/user/danbri/spectral/input/_topic_skm.csv<r 1>
494017428
grunt> mv _topic_skm.csv wikitopics.csv
grunt> quit
I should also describe the test data: it's a bipartite graph of affinities
between normal Wikipedia entries, and the Wikipedia categories that apply to
them. The weight is boolean 0 or 1, and is (on Shannon's advice) expressed in
both directions. So from the urldict.txt file, we see
0 http://dbpedia.org/resource/Albedo
1 http://dbpedia.org/resource/Category:Electromagnetic_radiation
2 http://dbpedia.org/resource/Category:Climatology
3 http://dbpedia.org/resource/Category:Climate_forcing
4
http://dbpedia.org/resource/Category:Scattering,_absorption_and_radiative_transfer_(optics)
5 http://dbpedia.org/resource/Category:Radiometry
which maps to
0,1,1.0
1,0,1.0
0,2,1.0
2,0,1.0
0,3,1.0
3,0,1.0
0,4,1.0
4,0,1.0
0,5,1.0
5,0,1.0
...which captures the associations between http://dbpedia.org/resource/Albedo
and the 5 topics assigned to it in Wikipedia/dbpedia.
Is this a good use of mahout's spectral clustering, specifically? Hard to say,
... didn't get past the entry gate yet. Maybe a more symmetric link graph
would've been a better first test.
See also
http://mail-archives.apache.org/mod_mbox/mahout-user/201203.mbox/%3CCAFfrAFrFMNvvXEaFy0rSZT1tA=S6pCW-+YHrOhR=w5hgv_v...@mail.gmail.com%3E
http://comments.gmane.org/gmane.comp.apache.mahout.user/12346 ... it would also
be interesting to try pulling second-smallest eigenvalue/vector from SSVD .
> OutOfMemoryError in LanczosState by way of SpectralKMeans
> ---------------------------------------------------------
>
> Key: MAHOUT-986
> URL: https://issues.apache.org/jira/browse/MAHOUT-986
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.6
> Environment: Ubuntu 11.10 (64-bit)
> Reporter: Shannon Quinn
> Assignee: Shannon Quinn
> Priority: Minor
> Fix For: 0.7
>
> Attachments: MAHOUT-986.patch
>
>
> Dan Brickley and I have been testing SpectralKMeans with a dbpedia dataset (
> http://danbri.org/2012/spectral/dbpedia/ ); effectively, a graph with
> 4,192,499 nodes. Not surprisingly, the LanczosSolver throws an
> OutOfMemoryError when it attempts to instantiate a DenseMatrix of dimensions
> 4192499-by-4192499 (~17.5 trillion double-precision floating point values).
> Here's the full stack trace:
> {quote}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at org.apache.mahout.math.DenseMatrix.<init>(DenseMatrix.java:50)
> at
> org.apache.mahout.math.decomposer.lanczos.LanczosState.<init>(LanczosState.java:45)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.run(SpectralKMeansDriver.java:146)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.run(SpectralKMeansDriver.java:86)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.mahout.clustering.spectral.kmeans.SpectralKMeansDriver.main(SpectralKMeansDriver.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> {quote}
> Obviously SKM needs a more sustainable and memory-efficient way of performing
> an eigen-decomposition of the graph laplacian. For those who are more
> knowledgeable in the linear systems solvers of Mahout than I, can the Lanczos
> parameters be tweaked to negate the requirement of a full DenseMatrix? Or
> should SKM move to SSVD instead?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
