[
https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175189#comment-13175189
]
Jeff Eastman commented on MAHOUT-846:
-------------------------------------
This version gets rid of all vector operations and caches the zProd2piS term
which is constant over the life of the cluster between reCalculations. I don't
really like the lazy evaluation but the alternative is putting it in the
constructors and reader. Will see...
{code}
// the value of the zProduct(S*2pi) term. Calculated below.
private Double zProd2piS = null;
@Override
public double pdf(VectorWritable vw) {
if (zProd2piS == null) {
computeProd2piS();
}
return Math.exp(-(sumXminusMdivSsquared(vw.get()) / 2)) / zProd2piS;
}
/**
* Compute the product(s[i]*SQRT2PI) over all i
*/
private void computeProd2piS() {
zProd2piS = 1.0;
for (Iterator<Element> it = getRadius().iterateNonZero(); it.hasNext();) {
Element se = it.next();
zProd2piS *= se.get() * UncommonDistributions.SQRT2PI;
}
}
@Override
public void computeParameters() {
super.computeParameters();
zProd2piS = null;
}
/**
* @param x
* a Vector
* @return the zSum(((x[i]-m[i])/s[i])^2) over i
*/
private double sumXminusMdivSsquared(Vector x) {
double result = 0;
for (Iterator<Element> it = getRadius().iterateNonZero(); it.hasNext();) {
Element se = it.next();
double quotient = (x.get(se.index()) - getCenter().get(se.index()))
/ se.get();
result += quotient * quotient;
}
return result;
}
{code}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a
> running product of the element pdfs which, for wide input vectors (Reuters is
> 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
