[
https://issues.apache.org/jira/browse/MAHOUT-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916435#action_12916435
]
Ankur commented on MAHOUT-344:
------------------------------
> Catching OutOfMemoryError
LastfmClusterEvaluator - I was getting OOME as I was trying to run this on
simply concatenated sequence files. Don't think we can proceed in this case as
simply concatenated sequence files essentially mean corrupt data. Once this was
fixed, there was no problem so we can safely remove the OOME catch block.
LastfmDataConverter - The code works fine with -Xmx512m heap settings that
should be reasonable for 1.5G of uncompressed data. This is what is suggested.
Can't think of a reasonable in-memory approach.
>Random - java.util.Random
Each mapper should get the exact copy of hash functions for constructing the
minhash signatures or else the chances of collision are quite less even for
highly similar items. that is the reason for hard-coding seed for Random(). The
reason for test failure is not that, it is the linear hash function that has a
higher false positive rate than murmur and polynomial hash functions. The
remedy is to concatenate more hashes for a key groups. For
testLinearMinHashMRJob() I changed number of hash functions to 20 and number of
key groups to 4 and now test passes successfully.
> Minhash based clustering
> -------------------------
>
> Key: MAHOUT-344
> URL: https://issues.apache.org/jira/browse/MAHOUT-344
> Project: Mahout
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Ankur
> Assignee: Ankur
> Fix For: 0.4
>
> Attachments: MAHOUT-344-v1.patch, MAHOUT-344-v2.patch,
> MAHOUT-344-v3.patch, MAHOUT-344-v4.patch, MAHOUT-344-v5.patch,
> MAHOUT-344-v6.patch
>
>
> Minhash clustering performs probabilistic dimension reduction of high
> dimensional data. The essence of the technique is to hash each item using
> multiple independent hash functions such that the probability of collision of
> similar items is higher. Multiple such hash tables can then be constructed
> to answer near neighbor type of queries efficiently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
