zhengruifeng commented on a change in pull request #26575: [SPARK-29942][ML]
Impl Complement Naive Bayes Classifier
URL: https://github.com/apache/spark/pull/26575#discussion_r347322235
##########
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
##########
@@ -204,14 +209,29 @@ class NaiveBayes @Since("1.5.0") (
val piArray = new Array[Double](numLabels)
val thetaArray = new Array[Double](numLabels * numFeatures)
+ val aggIter = $(modelType) match {
+ case Multinomial | Bernoulli => aggregated.iterator
+ case Complement =>
+ val featureSum = Vectors.zeros(numFeatures)
+ aggregated.foreach { case (_, _, sumTermFreqs, _) =>
+ BLAS.axpy(1.0, sumTermFreqs, featureSum)
+ }
+ aggregated.iterator.map { case (label, n, sumTermFreqs, count) =>
+ val comp = featureSum.copy
+ BLAS.axpy(-1.0, sumTermFreqs, comp)
+ (label, n, comp, count)
+ }
+ }
+
val lambda = $(smoothing)
val piLogDenom = math.log(numDocuments + numLabels * lambda)
var i = 0
- aggregated.foreach { case (label, n, sumTermFreqs, _) =>
+ aggIter.foreach { case (label, n, sumTermFreqs, _) =>
labelArray(i) = label
piArray(i) = math.log(n + lambda) - piLogDenom
val thetaLogDenom = $(modelType) match {
- case Multinomial => math.log(sumTermFreqs.toArray.sum + numFeatures *
lambda)
+ case Multinomial | Complement =>
+ math.log(sumTermFreqs.toArray.sum + numFeatures * lambda)
Review comment:
The logic of MNB can be reused for `Complement`, we only need to 1:compute
complement at first, and 2:make Matrx `theta=-theta` at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
