Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/4419#issuecomment-88872665
I tried it with 20_newsgroups on my dev PC with master=local
lda.setK(20).setMiniBatchFraction(0.02).setNumIterations(50).setTau_0(1)
Corpus summary:
Training set size: 11314 documents
Vocabulary size: 75720 terms
Training set size: 1319189 tokens
Preprocessing time: 5.269933486 sec
15/04/02 16:47:35 WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS
15/04/02 16:47:35 WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS
Finished training LDA model. Summary:
Training time: 76.832266468 sec
20 topics:
TOPIC 0
turkish 0.012058716788185058
armenians 0.011981509726531264
armenian 0.010004235150011833
armenia 0.006114108792354288
world 0.005581069826432254
women 0.005329639527608472
war 0.005310177119620838
turks 0.005239327705142824
turkey 0.00515853811651157
jason 0.005097117216929361
TOPIC 1
key 0.01316648107372953
chip 0.010541809566170054
government 0.010153778875200192
encryption 0.009326779907233067
clipper 0.008439466049444127
public 0.007530569438593113
server 0.007261400075345819
system 0.006687091520336047
technology 0.006480600341755369
information 0.006316195503698238
TOPIC 2
jews 0.009702004869705672
book 0.007853214980848604
law 0.0072516029853666045
jewish 0.006976932023763277
day 0.005739847653103481
children 0.005001342704817967
writes 0.00493066449176571
father 0.0046775540842062535
subject 0.004526731671596464
word 0.004261478063021758
TOPIC 3
lines 0.0119093867472742
window 0.011427052669067456
subject 0.011425847994190297
bike 0.011017175200925428
organization 0.010201230717622143
dod 0.010043030759882773
number 0.007815779704716674
writes 0.007217862379522699
comments 0.007134599945200453
jeff 0.006439084192687904
TOPIC 4
lines 0.012697140646928512
subject 0.012137529322560533
health 0.011682611293142414
organization 0.011528032382047063
child 0.009628843075919892
writes 0.0094157758574453
tax 0.009043295916954986
scott 0.008076554420454728
cramer 0.007101397643094552
uunet 0.00675596330842285
TOPIC 5
writes 0.00775952426885881
subject 0.006813048810540294
lines 0.006104476761875572
organization 0.005870115294116984
time 0.005372893507922081
make 0.005372502622392409
things 0.005244802597423591
space 0.004842330516924673
work 0.0041157380189356845
find 0.004003129079883294
TOPIC 6
god 0.03751706271018263
jesus 0.01809338668630806
bible 0.01273876849278002
christian 0.011515937636034564
truth 0.010207481599140669
christians 0.009734060715720241
true 0.009484256732270106
faith 0.009472189499476388
christ 0.008925357041998224
subject 0.008794316465052065
TOPIC 7
israel 0.012796192036701572
year 0.010460517189457452
israeli 0.009566052338340451
writes 0.00768129607686239
young 0.006786117523012472
subject 0.006212594045699227
player 0.006201122596017433
years 0.005983312056044822
lines 0.005965734811237885
organization 0.00588173227558923
TOPIC 8
organization 0.009066382601536617
writes 0.009034865041789575
subject 0.008994865803746694
lines 0.008685308635338337
good 0.007260135125978055
time 0.006539877322034039
kill 0.005511957186284815
long 0.005373592823169537
day 0.005142388760559978
made 0.0048908894669731995
TOPIC 9
file 0.02061426397921947
subject 0.01848451880060305
lines 0.018451157015520685
organization 0.017565345699419276
files 0.013872841520549063
university 0.011910930778210866
version 0.011640873128443674
mac 0.010282403737157388
program 0.010009092176160198
software 0.009597634369517759
TOPIC 10
writes 0.015805209959701905
organization 0.014876675058153706
subject 0.01470540557262385
lines 0.01344357702892899
bill 0.0095130114583992
texas 0.007206431218924633
michael 0.007178008874528052
moon 0.007036346761009455
university 0.00653090859458465
msg 0.006096418108324479
TOPIC 11
game 0.014674661401040233
team 0.013191146246910222
play 0.01010707995403076
hockey 0.00763938300173508
games 0.007147533054310146
league 0.006995082062115003
win 0.006886276134145244
nhl 0.006352039295683316
season 0.006324133614914862
year 0.006066287307435158
TOPIC 12
media 0.009886769239547982
number 0.007791276679198658
applications 0.007464461853157522
printer 0.0069284277659139625
slow 0.0063737660373687915
final 0.0063017559062341685
display 0.006202170194951673
font 0.005918182002639881
weapon 0.005859511598321789
line 0.005673207694267071
TOPIC 13
lines 0.019038844609056707
subject 0.01867889041065237
organization 0.01863154404976698
university 0.01052806373066347
modem 0.007296262423776374
insurance 0.006706286366160585
ohio 0.0055479057363888645
shipping 0.005144207558982205
keywords 0.004719857675933903
valid 0.004592917999173226
TOPIC 14
lines 0.027743859508015972
subject 0.027223348067573386
organization 0.02595495630938305
windows 0.024189290961090792
drive 0.017022504545593013
university 0.015605362805097134
card 0.014533215563853688
problem 0.011159934238387473
disk 0.010699944070108252
dos 0.009886396242470662
TOPIC 15
max 0.08874578618327673
air 0.01135975971362627
water 0.010048845884928805
ground 0.009358364099568165
wiring 0.008658287879779946
bhj 0.007680045983477506
hot 0.006288355248206904
giz 0.006246027066244181
grant 0.00524119552915761
bxn 0.004902311521825763
TOPIC 16
subject 0.013968368891644113
organization 0.013592128327318657
lines 0.013135179142445825
university 0.006997169480004322
drivers 0.006895162869223282
computer 0.006763309415528489
list 0.0066359312429406445
internet 0.006075399371871774
mail 0.006014434970537192
writes 0.005602600081269788
TOPIC 17
keyboard 0.01223302361505956
window 0.00898082252199221
motif 0.008967087211735314
clock 0.007854529186790353
subject 0.0076639431371468775
lines 0.007204139008145851
page 0.006974667674085922
library 0.0061012994381090205
organization 0.006094251981423491
application 0.00589426046208701
TOPIC 18
players 0.010997334837545092
subject 0.010768729154177278
lines 0.01076239370321377
organization 0.010695482778729407
speed 0.010097718273861734
writes 0.009870602880910849
wire 0.009020100414062154
car 0.008602098673758638
cars 0.007082965832769178
good 0.0069138751648267485
TOPIC 19
gun 0.01041215866197905
police 0.007151858743131414
weapons 0.006701168281112015
cover 0.006445776318889374
completely 0.006396634763223227
states 0.006286743675459915
law 0.006009817769080257
government 0.005870735254987818
rate 0.005717125865256983
crime 0.005551858992892133
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
