Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/10152#issuecomment-165093398
@ygcao thanks for making all the changes, this PR is a lot clearer now
focusing in on the core issue. I think you can also link
https://issues.apache.org/jira/browse/SPARK-7617 to this as your latest changes
also fixes that. Could you also perhaps update your JIRA description to reflect
the new scope of changes in this PR?
Pinging @jkbradley @mengxr @MechCoder - can you just double check that
you're happy with this - i.e. the core issue as I mentioned previously is that
the current implementation is incorrect and discards all sentence boundaries,
instead simply clipping "sentences" every 1000 words. This does not match the
original Google impl (as well as others e.g. Gensim and DL4J). These changes
now respect sentence boundaries in the input RDD, and imposes a max sentence
length (default the previous hard-coded `1000`).
Can we also whitelist to run the tests - I'm interested to see if this
impacts the Word2Vec tests in any way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
