[ https://issues.apache.org/jira/browse/SPARK-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695359#comment-15695359 ]
Sean Owen commented on SPARK-18581: ----------------------------------- It sounds like there's definitely a problem here but why is the pseudo-determinant the issue? it's possible for eigenvalues to be in (0,1) in which case the sum of logs is negative. Is that a problem per se? > MultivariateGaussian does not check if covariance matrix is invertible > ---------------------------------------------------------------------- > > Key: SPARK-18581 > URL: https://issues.apache.org/jira/browse/SPARK-18581 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.6.2, 2.0.2 > Reporter: Hao Ren > > When training GaussianMixtureModel, I found some probability much larger than > 1. That leads me to that fact that, the value returned by > MultivariateGaussian.pdf can be 10^5, etc. > After reviewing the code, I found that problem lies in the computation of > determinant of the covariance matrix. > The computation is simplified by using pseudo-determinant of a positive > defined matrix. However, if the eigen value is all between 0 and 1, > log(pseudo-determinant) will be a negative number like, -50. As a result, > the logpdf could be positive, thus pdf > 1 > The related code that the following: > // In function: MultivariateGaussian.calculateCovarianceConstants > {code} > val logPseudoDetSigma = d.activeValuesIterator.filter(_ > > tol).map(math.log).sum > {code} > d is the eigen value vector here. If lots of its elements are between 0 and > 1, then logPseudoDetSigma could be negative. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org