[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on GraphX
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-62077399 @witgo Thanks for the PR! This looks like a very featureful implementation, but I think it will require some refactoring to fit in well with future development. I'll give some high-level comments for now, and can perhaps do a lower-level pass later on. **APIs** I suspect we'll have other types of topic modeling in the future, not just LDA. It would be great to think ahead for that. The simplest way is probably to rename everything as LDA, not topic modeling, and to minimize the public API. (Other topic models we might want later are LSA, PLSA, HDP, CTM, etc.) This should probably go under clustering instead of feature. **Code organization** Some of the code is more general than LDA and could go elsewhere in MLlib. E.g., some of the sampling methods could go in stat/ Also, minMaxIndexSearch, minMaxValueSearch, etc. (or can those be replaced using existing generic methods in Scala or Java?). **Documentation and code clarity** The current thing making this hardest to review is the lack of documentation and the difficulty in understanding what each value and method does. For documentation, it will be helpful to see comments for all classes and methods, and also inline comments explaining code where needed. For code clarity, using more descriptive variable and method names will help a lot. **Other thoughts** It would be nice to remove some experimental items (such as mergeDuplicateTopic) for now. Thanks again! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on GraphX
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-62077644 [Test build #512 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/512/consoleFull) for PR 2388 at commit [`fe40445`](https://github.com/apache/spark/commit/fe404451e1d7cf15a336160732a5d2a94c99a877). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on GraphX
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-62084868 [Test build #512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/512/consoleFull) for PR 2388 at commit [`fe40445`](https://github.com/apache/spark/commit/fe404451e1d7cf15a336160732a5d2a94c99a877). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-60951854 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58856594 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21683/consoleFull) for PR 2388 at commit [`1e2485c`](https://github.com/apache/spark/commit/1e2485c05c77dbca4332b9af616c27c45f2f5e32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58862844 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21683/consoleFull) for PR 2388 at commit [`1e2485c`](https://github.com/apache/spark/commit/1e2485c05c77dbca4332b9af616c27c45f2f5e32). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TopicModelingKryoRegistrator extends KryoRegistrator ` * `class StreamingContext(object):` * `class DStream(object):` * `class TransformedDStream(DStream):` * `class TransformFunction(object):` * `class TransformFunctionSerializer(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58862848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21683/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/2388#discussion_r18768316
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/feature/TopicModeling.scala ---
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector = BDV, SparseVector = BSV, sum =
brzSum}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.broadcast.Broadcast
+import org.apache.spark.graphx._
+import org.apache.spark.Logging
+import org.apache.spark.mllib.linalg.distributed.{MatrixEntry, RowMatrix}
+import org.apache.spark.mllib.linalg.{DenseVector = SDV, SparseVector =
SSV, Vector = SV}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.serializer.KryoRegistrator
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.SparkContext._
+
+import TopicModeling._
+
+class TopicModeling private[mllib](
+ @transient var corpus: Graph[VD, ED],
+ val numTopics: Int,
+ val numTerms: Int,
+ val alpha: Double,
+ val beta: Double,
+ @transient val storageLevel: StorageLevel)
+ extends Serializable with Logging {
+
+ def this(docs: RDD[(TopicModeling.DocId, SSV)],
+numTopics: Int,
+alpha: Double,
+beta: Double,
+storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK,
+computedModel: Broadcast[TopicModel] = null) {
+this(initializeCorpus(docs, numTopics, storageLevel, computedModel),
+ numTopics, docs.first()._2.size, alpha, beta, storageLevel)
+ }
+
+
+ /**
+ * The number of documents in the corpus
+ */
+ val numDocs = docVertices.count()
+
+ /**
+ * The number of terms in the corpus
+ */
+ private val sumTerms = corpus.edges.map(e =
e.attr.size.toDouble).sum().toLong
+
+ /**
+ * The total counts for each topic
+ */
+ @transient private var globalTopicCounter: BDV[Count] =
collectGlobalCounter(corpus, numTopics)
+ assert(brzSum(globalTopicCounter) == sumTerms)
+
+ @transient private val sc = corpus.vertices.context
+ @transient private val seed = new Random().nextInt()
+ @transient private var innerIter = 1
+ @transient private var cachedEdges: EdgeRDD[ED, VD] = corpus.edges
+ @transient private var cachedVertices: VertexRDD[VD] = corpus.vertices
+
+ private def termVertices = corpus.vertices.filter(t = t._1 = 0)
+
+ private def docVertices = corpus.vertices.filter(t = t._1 0)
+
+ private def checkpoint(): Unit = {
+if (innerIter % 10 == 0 sc.getCheckpointDir.isDefined) {
+ val edges = corpus.edges.map(t = t)
+ edges.checkpoint()
+ val newCorpus: Graph[VD, ED] = Graph.fromEdges(edges, null,
+storageLevel, storageLevel)
+ corpus = updateCounter(newCorpus, numTopics).cache()
+}
+ }
+
+ private def gibbsSampling(): Unit = {
+val corpusTopicDist = collectTermTopicDist(corpus, globalTopicCounter,
+ sumTerms, numTerms, numTopics, alpha, beta)
+
+val corpusSampleTopics = sampleTopics(corpusTopicDist,
globalTopicCounter,
+ sumTerms, innerIter + seed, numTerms, numTopics, alpha, beta)
+corpusSampleTopics.edges.setName(sedges-$innerIter).cache().count()
+Option(cachedEdges).foreach(_.unpersist())
+cachedEdges = corpusSampleTopics.edges
+
+corpus = updateCounter(corpusSampleTopics, numTopics)
+corpus.vertices.setName(svertices-$innerIter).cache()
+globalTopicCounter = collectGlobalCounter(corpus, numTopics)
+assert(brzSum(globalTopicCounter) == sumTerms)
+Option(cachedVertices).foreach(_.unpersist())
+cachedVertices = corpus.vertices
+
+checkpoint()
+innerIter += 1
+ }
+
+ def saveTopicModel(burnInIter: Int): TopicModel = {
+val topicModel =
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58744928 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21642/consoleFull) for PR 2388 at commit [`b0734b8`](https://github.com/apache/spark/commit/b0734b86ab95774aec79af55d9de48b363fe243b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/2388#discussion_r18740868
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/feature/TopicModeling.scala ---
@@ -0,0 +1,674 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector = BDV, SparseVector = BSV, Vector =
BV, sum = brzSum}
+
+import org.apache.spark.Logging
+import org.apache.spark.SparkContext._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.broadcast.Broadcast
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.distributed.{MatrixEntry, RowMatrix}
+import org.apache.spark.mllib.linalg.{DenseVector = SDV, SparseVector =
SSV, Vector = SV}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.serializer.KryoRegistrator
+import org.apache.spark.storage.StorageLevel
+
+import TopicModeling._
+
+class TopicModeling private[mllib](
+ @transient var corpus: Graph[VD, ED],
+ val numTopics: Int,
+ val numTerms: Int,
+ val alpha: Double,
+ val beta: Double,
+ @transient val storageLevel: StorageLevel)
+ extends Serializable with Logging {
+
+ def this(docs: RDD[(TopicModeling.DocId, SSV)],
+numTopics: Int,
+alpha: Double,
+beta: Double,
+storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK,
+computedModel: Broadcast[TopicModel] = null) {
+this(initializeCorpus(docs, numTopics, storageLevel, computedModel),
+ numTopics, docs.first()._2.size, alpha, beta, storageLevel)
+ }
+
+
+ /**
+ * The number of documents in the corpus
+ */
+ val numDocs = docVertices.count()
+
+ /**
+ * The number of terms in the corpus
+ */
+ private val sumTerms = corpus.edges.map(e =
e.attr.size.toDouble).sum().toLong
+
+ /**
+ * The total counts for each topic
+ */
+ @transient private var globalTopicCounter: BV[Count] =
collectGlobalCounter(corpus, numTopics)
+ assert(brzSum(globalTopicCounter) == sumTerms)
+ @transient private val sc = corpus.vertices.context
+ @transient private val seed = new Random().nextInt()
+ @transient private var innerIter = 1
+ @transient private var cachedEdges: EdgeRDD[ED, VD] = null
+ @transient private var cachedVertices: VertexRDD[VD] = null
+
+ private def termVertices = corpus.vertices.filter(t = t._1 = 0)
+
+ private def docVertices = corpus.vertices.filter(t = t._1 0)
+
+ private def gibbsSampling(cachedEdges: EdgeRDD[ED, VD],
+cachedVertices: VertexRDD[VD]): (EdgeRDD[ED, VD], VertexRDD[VD]) = {
+
+val corpusTopicDist = collectTermTopicDist(corpus, globalTopicCounter,
+ sumTerms, numTerms, numTopics, alpha, beta)
+
+val corpusSampleTopics = sampleTopics(corpusTopicDist,
globalTopicCounter,
+ sumTerms, innerIter + seed, numTerms, numTopics, alpha, beta)
+corpusSampleTopics.edges.setName(sedges-$innerIter).cache().count()
+Option(cachedEdges).foreach(_.unpersist())
+val edges = corpusSampleTopics.edges
+
+corpus = updateCounter(corpusSampleTopics, numTopics)
+corpus.vertices.setName(svertices-$innerIter).cache()
+globalTopicCounter = collectGlobalCounter(corpus, numTopics)
+assert(brzSum(globalTopicCounter) == sumTerms)
+Option(cachedVertices).foreach(_.unpersist())
+val vertices = corpus.vertices
+
+if (innerIter % 10 == 0 sc.getCheckpointDir.isDefined) {
--- End diff --
This is only a temporary solution.
The related PR #2631
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58746403 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21642/consoleFull) for PR 2388 at commit [`b0734b8`](https://github.com/apache/spark/commit/b0734b86ab95774aec79af55d9de48b363fe243b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TopicModelingKryoRegistrator extends KryoRegistrator ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58746407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21642/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58324547 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21455/consoleFull) for PR 2388 at commit [`ca8e6f2`](https://github.com/apache/spark/commit/ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58330999 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21455/consoleFull) for PR 2388 at commit [`ca8e6f2`](https://github.com/apache/spark/commit/ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class Params(inputFile: String = null, threshold: Double = 0.1)` * `class TopicModelingKryoRegistrator extends KryoRegistrator ` * `class Word2VecModel(object):` * `class Word2Vec(object):` * ` class SparkIMain(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58331009 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21455/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
