[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-55956 I think the original Map was OK, IMHO. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user acidghost commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-60588 @srowen I just created PR #6761 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user acidghost commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-31476 @jkbradley @srowen Do you think that the return type of `predictProbabilities` should be `scala.collection.Map[Double, Double]` or `Vector[Double]`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user acidghost commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-110630408 Hello there! I'm interested in reopening this PR and contributing with a patch to [SPARK-4362] based on this PR with the needed changes. I read [the wiki](https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark), should I simply follow those steps and create a new PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3626 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-78520891 Mind closing this PR? if it's not going to be updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-75564514 @alanctgardner have you had a look at @jkbradley 's feedback? I'm wondering this is still live. It needs a rebase if so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user alanctgardner commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r25172516
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +66,25 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
--- End diff --
Sorry for the delay, I have no strong preference but predictProbabilities
makes sense for consistency. I can make that change and the style ones
mentioned.
My stats background is not super-strong, @jatinpreet seemed to imply
there's a correctness issue with this PR. Can anyone comment on if I've got the
math wrong?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-75608187 @alanctgardner That will be great if you change it to predictProbabilities; thanks. I agree with what @jatinpreet was saying about the correctness, and with @srowen 's comment on how to fix it: The value of ```brzPi + brzTheta * testData.toBreeze``` is a log probability, which needs to be exponentiated before you normalize it here: [https://github.com/apache/spark/pull/3626/files?diff=split#diff-6d8eff78be2fb624d4a076db334208a4R84] Could you please rebase off of master and make these couple of updates? After that, I can make a final pass. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21563627
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +66,25 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector): scala.collection.Map[Double,
Double] = {
+val posteriors = (brzPi + brzTheta * testData.toBreeze)
+val sum = posteriors.sum
+val probs:mutable.Map[Double,Double] =
+ mutable.Map.empty[Double, Double]
+posteriors.foreachPair((k,v) = probs += (labels(k) - v/sum))
--- End diff --
scala style: space after comma + space around / operator
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21563623
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +66,25 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector): scala.collection.Map[Double,
Double] = {
+val posteriors = (brzPi + brzTheta * testData.toBreeze)
+val sum = posteriors.sum
+val probs:mutable.Map[Double,Double] =
+ mutable.Map.empty[Double, Double]
--- End diff --
This can fit on 1 line.
Also, there's no need to specify the type of probs since
mutable.Map.empty[Double, Double] makes it clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21564191
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +66,25 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
--- End diff --
Classes are index from 0, so would it work to return (for each instance) a
Vector of probabilities rather than a Map[Double, Double]? That seems simpler
(and more efficient).
Also, I have a PR [https://github.com/apache/spark/pull/3637] for a
separate part of MLlib which adds a method like this for predicting class
conditional probabilities. I'd like us to use the same name for the prediction
method, but I'm open about choosing a name. I had used predictProbabilities
to (a) have it start with predict like other prediction methods and (b) leave
open the possibility of supporting a similar method for regression algorithms
(which can predict probability distributions). But I'll agree
classProbabilities is more specific. Do you have strong preferences?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
GitHub user alanctgardner opened a pull request: https://github.com/apache/spark/pull/3626 [MLLIB] SPARK-4362: Added classProbabilities method for Naive Bayes Added methods which accept an RDD or array and return a map of (label - posterior prob.) for each input set rather than only returning the key with the maximum value. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alanctgardner/spark nb-posterior Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3626.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3626 commit da4fddaeaa6a4c72e6024db1df1ff7d1a356ff90 Author: Alan Gardner alanctgard...@gmail.com Date: 2014-12-05T21:10:31Z Added classProbabilities method --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21402613
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
--- End diff --
I'd probably import `scala.collection.mutable` and write `mutable.Map`. Not
sure what others' preference is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3626#issuecomment-65859729 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21402733
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector):
scala.collection.mutable.Map[Double, Double] = {
--- End diff --
.. but why are you returning mutable `Map` anyway?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21402810
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector):
scala.collection.mutable.Map[Double, Double] = {
+val posteriors = (brzPi + brzTheta * testData.toBreeze)
--- End diff --
These aren't quite the class probabilities as the expression doesn't have
the probability of the evidence incorporated. This would work if you first
normalized the probabilities to sum to 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user alanctgardner commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21402907
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector):
scala.collection.mutable.Map[Double, Double] = {
--- End diff --
Scala newbie. I couldn't find a better pattern to build the map than
mutating it in the foreach. Should I just build a map then make it immutable
for returning?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-4362: Added classProbabilities m...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3626#discussion_r21403159
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector):
scala.collection.mutable.Map[Double, Double] = {
--- End diff --
That's fine, but you need not promise a mutable Map in the return
type. You can return it as a scala.collection.Map
On Fri, Dec 5, 2014 at 3:51 PM, alanctgardner notificati...@github.com
wrote:
In
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:
@@ -65,6 +65,24 @@ class NaiveBayesModel private[mllib] (
override def predict(testData: Vector): Double = {
labels(brzArgmax(brzPi + brzTheta * testData.toBreeze))
}
+
+ def classProbabilities(testData: RDD[Vector]):
+ RDD[scala.collection.mutable.Map[Double, Double]] = {
+val bcModel = testData.context.broadcast(this)
+testData.mapPartitions { iter =
+ val model = bcModel.value
+ iter.map(model.classProbabilities)
+}
+ }
+
+ def classProbabilities(testData: Vector):
scala.collection.mutable.Map[Double, Double] = {
Scala newbie. I couldn't find a better pattern to build the map than
mutating it in the foreach. Should I just build a map then make it
immutable
for returning?
â
Reply to this email directly or view it on GitHub.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
