[
https://issues.apache.org/jira/browse/SPARK-18374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710198#comment-15710198
]
yuhao yang commented on SPARK-18374:
------------------------------------
I checked with some other lists of stopwords and got a list to add, but it's
longer than I thought:
i'll
you'll
he'll
she'll
we'll
they'll
i'd
you'd
he'd
she'd
we'd
they'd
i'm
you're
he's
she's
it's
we're
they're
i've
we've
you've
they've
isn't
aren't
wasn't
weren't
haven't
hasn't
hadn't
don't
doesn't
didn't
won't
wouldn't
shan't
shouldn't
mustn't
can't
couldn't
any concern?
> Incorrect words in StopWords/english.txt
> ----------------------------------------
>
> Key: SPARK-18374
> URL: https://issues.apache.org/jira/browse/SPARK-18374
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.0.1
> Reporter: nirav patel
>
> I was just double checking english.txt for list of stopwords as I felt it was
> taking out valid tokens like 'won'. I think issue is english.txt list is
> missing apostrophe character and all character after apostrophe. So "won't"
> becam "won" in that list; "wouldn't" is "wouldn" .
> Here are some incorrect tokens in this list:
> won
> wouldn
> ma
> mightn
> mustn
> needn
> shan
> shouldn
> wasn
> weren
> I think ideal list should have both style. i.e. won't and wont both should be
> part of english.txt as some tokenizer might remove special characters. But
> 'won' is obviously shouldn't be in this list.
> Here's list of snowball english stop words:
> http://snowball.tartarus.org/algorithms/english/stop.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
