Sorry, forgot the test change, here's the full patch.
2015-02-19 18:58 GMT-05:00 Andriy Rysin <ary...@gmail.com>:
> Daniel
>
> I took a look at the problem of SameRuleGroupFilter missing rules on
> multithreaded execution due to rules with same id being split across
> threads. So I've added a SameRuleGroupFilter.filter() after all
> threads return.
> But to my surprise the tests that compare single-threaded run with
> multithreaded started to fail and by debugging results I could see
> that the new multithreaded run is correct.
> So I had to add a fix for single-threaded run by adding
> SameRuleGroupFilter after sort().
>
> I don't know all the details so I am attaching the patch rather than
> pusing it in, could you please take a look?
>
> Also with this we run SameRuleGroupFilter twice for both modes - one
> time (per thread) inside performCheck() and once at the end of check()
> after sorting. I feel like it's redundant and we can remove the first
> one.
>
> Thanks
> Andriy
>
>
>
>
> 2015-02-19 16:16 GMT-05:00 Andriy Rysin <ary...@gmail.com>:
>> I've merged multithreading branch into master. Please try it out when
>> you have a chance and let me know if you see any issues.
>>
>> Thanks
>> Andriy
>>
>> 2015-02-18 14:10 GMT-05:00 Andriy Rysin <ary...@gmail.com>:
>>> That makes sense, change pushed.
>>>
>>> Andriy
>>>
>>> 2015-02-18 11:48 GMT-05:00 Daniel Naber <daniel.na...@languagetool.org>:
>>>> On 2015-02-18 15:19, Andriy Rysin wrote:
>>>>
>>>>> 2) remove it completely, but I think it would be nice to have in case
>>>>> somebody (maybe me again :)) will want to do more performance
>>>>> profiling
>>>>
>>>> What about changing the property name to something that makes clear it's
>>>> internal any nobody should rely on it? Like
>>>> 'org.languagetool.thread_count_internal'?
>>>>
>>>> Regards
>>>> Daniel
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Languagetool-devel mailing list
>>>> Languagetool-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/languagetool-devel
diff --git a/languagetool-core/src/main/java/org/languagetool/JLanguageTool.java b/languagetool-core/src/main/java/org/languagetool/JLanguageTool.java
index 10ee9bb..956dfa9 100644
--- a/languagetool-core/src/main/java/org/languagetool/JLanguageTool.java
+++ b/languagetool-core/src/main/java/org/languagetool/JLanguageTool.java
@@ -479,9 +479,12 @@
unknownWords = new HashSet<>();
final List<AnalyzedSentence> analyzedSentences = analyzeSentences(sentences);
- final List<RuleMatch> ruleMatches = performCheck(analyzedSentences, sentences, allRules, paraMode, annotatedText);
+ List<RuleMatch> ruleMatches = performCheck(analyzedSentences, sentences, allRules, paraMode, annotatedText);
Collections.sort(ruleMatches);
+
+ ruleMatches = new SameRuleGroupFilter().filter(ruleMatches);
+
return ruleMatches;
}
diff --git a/languagetool-core/src/main/java/org/languagetool/MultiThreadedJLanguageTool.java b/languagetool-core/src/main/java/org/languagetool/MultiThreadedJLanguageTool.java
index e08de3d..d37639d 100644
--- a/languagetool-core/src/main/java/org/languagetool/MultiThreadedJLanguageTool.java
+++ b/languagetool-core/src/main/java/org/languagetool/MultiThreadedJLanguageTool.java
@@ -31,6 +31,7 @@
import org.languagetool.markup.AnnotatedText;
import org.languagetool.rules.Rule;
import org.languagetool.rules.RuleMatch;
+import org.languagetool.rules.SameRuleGroupFilter;
/**
* A variant of {@link JLanguageTool} that uses several threads for rule matching.
@@ -139,7 +140,7 @@
int lineCount = 0;
int columnCount = 1;
- final List<RuleMatch> ruleMatches = new ArrayList<>();
+ List<RuleMatch> ruleMatches = new ArrayList<>();
final ExecutorService executorService = getExecutorService();
try {
@@ -149,6 +150,8 @@
for (Future<List<RuleMatch>> future : futures) {
ruleMatches.addAll(future.get());
}
+
+ ruleMatches = new SameRuleGroupFilter().filter(ruleMatches);
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
diff --git a/languagetool-core/src/test/java/org/languagetool/MultiThreadedJLanguageToolTest.java b/languagetool-core/src/test/java/org/languagetool/MultiThreadedJLanguageToolTest.java
index a244dcb..12308d8 100644
--- a/languagetool-core/src/test/java/org/languagetool/MultiThreadedJLanguageToolTest.java
+++ b/languagetool-core/src/test/java/org/languagetool/MultiThreadedJLanguageToolTest.java
@@ -18,7 +18,6 @@
*/
package org.languagetool;
-import static junit.framework.TestCase.assertTrue;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;
@@ -31,7 +30,10 @@
import org.junit.Assert;
import org.junit.Test;
import org.languagetool.language.Demo;
-import org.languagetool.rules.*;
+import org.languagetool.rules.MultipleWhitespaceRule;
+import org.languagetool.rules.Rule;
+import org.languagetool.rules.RuleMatch;
+import org.languagetool.rules.UppercaseSentenceStartRule;
public class MultiThreadedJLanguageToolTest {
@@ -41,7 +43,7 @@
tool = new MultiThreadedJLanguageTool(new Demo());
final List<String> ruleMatchIds1 = getRuleMatchIds(tool);
- assertTrue(ruleMatchIds1.size() == 10);
+ Assert.assertEquals(9, ruleMatchIds1.size());
Assert.assertEquals(4, tool.getSentenceCount());
tool = new JLanguageTool(new Demo());------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Languagetool-devel mailing list
Languagetool-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/languagetool-devel
