viirya commented on issue #26722: [SPARK-24666][ML] Fix infinity vectors produced by Word2Vec when numIterations are large URL: https://github.com/apache/spark/pull/26722#issuecomment-560247040 > What I don't really understand is why it would be 'triggered' by the number of partitions rather than iterations here, or why it doesn't seem to show up otherwise. It's possible that it's really iterations driving this, and numPartitions isn't 'helping. Both the number of partitions and the iterations do affect to that. By reducing iteration to 10 (30 previously), when number of partitions is 5, you won't see infinity magnitude as iteration 30 case. ``` Training model..., numParts = 5 word: Martha's, magnitude: 3256804.914837854 word: Marta, magnitude: 353497.2433499305 word: Marvel's, magnitude: 4069358.2203939725 word: Arlovski, magnitude: 7118886.591379862 word: Nation:, magnitude: 6296374.743896999 word: Stock, magnitude: 4837719.561042786 word: #9:, magnitude: 2.2051319577420667E7 word: Chayon-Ryu, magnitude: 2965298.379611738 word: (Fifth, magnitude: 1.3429820237455452E7 word: Shiver, magnitude: 319441.2914574445 word: Porcupine, magnitude: 2267350.3638493987 word: Whiteman, magnitude: 260164.35322311163 word: Baldpate, magnitude: 2392710.378421927 word: Einstein, magnitude: 4124620.3929244205 word: Neapolitan, magnitude: 2840700.767267119 word: Vi, magnitude: 4.443334263321898 word: Tallest, magnitude: 285155.5317927394 word: Novak, magnitude: 2646500.711022009 word: Park', magnitude: 387377.63594714657 word: #28:, magnitude: 40106.517375608666 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
