[
https://issues.apache.org/jira/browse/MAHOUT-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966352#action_12966352
]
Dmitriy Lyubimov commented on MAHOUT-376:
-----------------------------------------
{quote}That means that we are limited to cases with a few hundred million
non-zero elements and are effectively unlimited on the number of potential
columns of A.{quote}
I beleive in case of SSVD this statement only partially valid.
It all depends on what you are spec'd to. say we are spec'd to 1G + java/mr
overhead, and few hundred million non-zero elements will take few hundred
megabytes multiplied by 8. Which is already all or more than all i have. In my
spec, (million non-zeros) it's only 8 mb that's seems ok. SSVD assumption
doesn't include any significant memory allocation for rows of A, and most
importantly, it doesn't have to, i think. Philosophy here is that A is a file,
and read it with a buffer to optimize I/O, but my stream buffer doesn't have to
be forced to be 1G on me.
> Implement Map-reduce version of stochastic SVD
> ----------------------------------------------
>
> Key: MAHOUT-376
> URL: https://issues.apache.org/jira/browse/MAHOUT-376
> Project: Mahout
> Issue Type: Improvement
> Components: Math
> Reporter: Ted Dunning
> Assignee: Ted Dunning
> Fix For: 0.5
>
> Attachments: MAHOUT-376.patch, Modified stochastic svd algorithm for
> mapreduce.pdf, QR decomposition for Map.pdf, QR decomposition for Map.pdf, QR
> decomposition for Map.pdf, sd-bib.bib, sd.pdf, sd.pdf, sd.pdf, sd.pdf,
> sd.tex, sd.tex, sd.tex, sd.tex, SSVD working notes.pdf, SSVD working
> notes.pdf, SSVD working notes.pdf, ssvd-CDH3-or-0.21.patch.gz,
> ssvd-CDH3-or-0.21.patch.gz, ssvd-m1.patch.gz, ssvd-m2.patch.gz,
> ssvd-m3.patch.gz, Stochastic SVD using eigensolver trick.pdf
>
>
> See attached pdf for outline of proposed method.
> All comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
