zhengruifeng commented on a change in pull request #27519:
[SPARK-30770][ML][WIP] avoid vector conversion in GMM.transform
URL: https://github.com/apache/spark/pull/27519#discussion_r384252384
##########
File path:
mllib-local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
##########
@@ -48,43 +48,37 @@ class MultivariateGaussian @Since("2.0.0") (
this(Vectors.fromBreeze(mean), Matrices.fromBreeze(cov))
}
- @transient private lazy val breezeMu = mean.asBreeze.toDenseVector
-
/**
* Compute distribution dependent constants:
* rootSigmaInv = D^(-1/2)^ * U.t, where sigma = U * D * U.t
* u = log((2*pi)^(-k/2)^ * det(sigma)^(-1/2)^)
*/
- @transient private lazy val tuple = calculateCovarianceConstants
- @transient private lazy val rootSigmaInv = tuple._1
+ @transient private lazy val tuple = {
+ val (rootSigmaInv, u) = calculateCovarianceConstants
+ val rootSigmaInvMat = Matrices.fromBreeze(rootSigmaInv)
+ (rootSigmaInvMat, u)
+ }
+ @transient private lazy val rootSigmaInvMat = tuple._1
@transient private lazy val u = tuple._2
+ @transient private lazy val mu = mean.toDense
/**
* Returns density of this multivariate Gaussian at given point, x
*/
@Since("2.0.0")
def pdf(x: Vector): Double = {
- pdf(x.asBreeze)
+ math.exp(logpdf(x))
Review comment:
Yes, that is what existing impl does:
```scala
/**
* Returns density of this multivariate Gaussian at given point, x
*/
@Since("2.0.0")
def pdf(x: Vector): Double = {
pdf(x.asBreeze)
}
/**
* Returns the log-density of this multivariate Gaussian at given point, x
*/
@Since("2.0.0")
def logpdf(x: Vector): Double = {
logpdf(x.asBreeze)
}
/** Returns density of this multivariate Gaussian at given point, x */
private[ml] def pdf(x: BV[Double]): Double = {
math.exp(logpdf(x))
}
/** Returns the log-density of this multivariate Gaussian at given point,
x */
private[ml] def logpdf(x: BV[Double]): Double = {
val delta = x - breezeMu
val v = rootSigmaInv * delta
u + v.t * v * -0.5
}
```
After this change, we do not need private methods based on BreezeVector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
