GitHub user witgo reopened a pull request:
https://github.com/apache/spark/pull/2388
[WIP][SPARK-1405][MLLIB] topic modeling on Graphx
This PR relies on #2631
- [X] Topic de-duplication
- [X] Support 100000 topics
- [X] Asymmetric Dirichlet priors
- [ ] Add the documentation
- [X] Add infer interface
- [X] Add unit tests
- [X] Add the performance test
- [X] Optimizing the infer interface performance
- [ ] Verifying the correctness of the algorithm
The performance test:
`2000` topics:
Item | value
------------ | -------------
The cluster resource | 36 executors(36 cores, 216g memory)
The corpus size | 253064 document, 29696335 words
The number of iterations | `105`
The number of distinct term | 75496
The number of topics | `2000`
alpha | 0.01
beta | 0.01
The running time | 37.1 minutes
`10000` topics:
Item | value
------------ | -------------
The cluster resource | 36 executors(36 cores, 216g memory)
The corpus size | 253064 document, 29696335 words
The number of iterations | `105`
The number of distinct term | 75496
The number of topics | `10000`
alpha | 0.01
beta | 0.01
The running time | 49 minutes
`100000` topics:
Item | value
------------ | -------------
The cluster resource | 36 executors(36 cores, 216g memory)
The corpus size | 253064 document, 29696335 words
The number of iterations | `105`
The number of distinct term | 75496
The number of topics | `100000`
alpha | 0.1
beta | 0.01
The running time | 268.9 minutes
conf/spark-defaults.conf:
```
spark.akka.frameSize 20
spark.executor.instances 36
spark.rdd.compress true
spark.executor.memory 6g
spark.default.parallelism 72
spark.broadcast.blockSize 8192
spark.storage.memoryFraction 0.4
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator
org.apache.spark.mllib.feature.TopicModelingKryoRegistrator
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/witgo/spark graphx_lda
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2388.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2388
----
commit ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14
Author: GuoQiang Li <[email protected]>
Date: 2014-10-08T08:10:12Z
topic modeling on Graphx
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
