viirya commented on issue #26722: [SPARK-24666][ML] Fix infinity vectors produced by Word2Vec when numIterations are large URL: https://github.com/apache/spark/pull/26722#issuecomment-560145783 This is the magnitude: ``` Training model..., numParts = 1 word: Martha's, magnitude: 3.743278710659449 word: Marta, magnitude: 3.0524119611411527 word: Marvel's, magnitude: 3.7116662524570962 word: Arlovski, magnitude: 5.367418309208839 word: Nation:, magnitude: 2.3689783957491817 word: Stock, magnitude: 3.6421844678790065 word: #9:, magnitude: 3.592700041933447 word: Chayon-Ryu, magnitude: 4.190948205327908 word: (Fifth, magnitude: 5.577325501834904 word: Shiver, magnitude: 2.924427895961321 word: Porcupine, magnitude: 2.979338865483149 word: Whiteman, magnitude: 2.9768204064161585 word: Baldpate, magnitude: 4.443153970183657 word: Einstein, magnitude: 2.534842496715387 word: Neapolitan, magnitude: 3.364223370424205 word: Vi, magnitude: 1.6161103277786248 word: Tallest, magnitude: 2.959881355124488 word: Novak, magnitude: 3.6077390984642848 word: Park', magnitude: 3.699203539517963 word: #28:, magnitude: 3.9433840429206155 ``` ``` Training model..., numParts = 5 word: Martha's, magnitude: Infinity word: Marta, magnitude: 3.8229410743441402E17 word: Marvel's, magnitude: Infinity word: Arlovski, magnitude: Infinity word: Nation:, magnitude: Infinity word: Stock, magnitude: Infinity word: #9:, magnitude: Infinity word: Chayon-Ryu, magnitude: Infinity word: (Fifth, magnitude: Infinity word: Shiver, magnitude: 3.4865850155309619E17 word: Porcupine, magnitude: Infinity word: Whiteman, magnitude: 2.9038712510688166E17 word: Baldpate, magnitude: Infinity word: Einstein, magnitude: Infinity word: Neapolitan, magnitude: Infinity word: Vi, magnitude: 3.476331244503146 word: Tallest, magnitude: 3.243632130386425E17 word: Novak, magnitude: Infinity word: Park', magnitude: 4.3857814193649882E17 word: #28:, magnitude: 1.442267607046858E14 ``` ``` Training model..., numParts = 50 word: Martha's, magnitude: Infinity word: Marta, magnitude: Infinity word: Marvel's, magnitude: Infinity word: Arlovski, magnitude: Infinity word: Nation:, magnitude: Infinity word: Stock, magnitude: Infinity word: #9:, magnitude: Infinity word: Chayon-Ryu, magnitude: Infinity word: (Fifth, magnitude: Infinity word: Shiver, magnitude: Infinity word: Porcupine, magnitude: Infinity word: Whiteman, magnitude: Infinity word: Baldpate, magnitude: Infinity word: Einstein, magnitude: Infinity word: Neapolitan, magnitude: Infinity word: Vi, magnitude: 2.313728521691476 word: Tallest, magnitude: Infinity word: Novak, magnitude: Infinity word: Park', magnitude: Infinity word: #28:, magnitude: Infinity ``` If divided by the number of partitions when aggregating weight vectors: ``` Training model..., numParts = 50 word: Martha's, magnitude: 1837.5938071293122 word: Marta, magnitude: 222.50718913947478 word: Marvel's, magnitude: 26457.30749717363 word: Arlovski, magnitude: 38.445068805110594 word: Nation:, magnitude: 965137.5871076621 word: Stock, magnitude: 7012249.419810451 word: #9:, magnitude: 2.0486049390084442E8 word: Chayon-Ryu, magnitude: 45.13060068472458 word: (Fifth, magnitude: 202.46650957627534 word: Shiver, magnitude: 200.8438739886034 word: Porcupine, magnitude: 78.98423745042425 word: Whiteman, magnitude: 17.939542328473102 word: Baldpate, magnitude: 36.798928419350744 word: Einstein, magnitude: 6474.188443349482 word: Neapolitan, magnitude: 607.5522611265635 word: Vi, magnitude: 6.211080604679997 word: Tallest, magnitude: 23.01892240254132 word: Novak, magnitude: 416.79496982929146 word: Park', magnitude: 29.33652629236024 word: #28:, magnitude: 29.42300764548468 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
