So before wrapping these optimizations up I decided to take a last
look at the thread graph in jvisualvm and it showed that the worker
threads spend more time in park state then in running. But the graph
was really not showing why, it was more like a noodle soup. So I
brought one of my past optimization back in: to always read file in
big blocks (don't start analyze/check on each paragraph break), this
made the thread graph very clear: besides waiting for main thread to
prepare sentences the check threads run times were not equal (we had
equal amount of rules per thread which does not actually amount to the
same load). So I've added another of my test optimizations which
didn't help before: creating a callable for each rule rather than for
group of rules.
The result: my cpu idle state went from 40% to 10% (now pretty much
all of those 10% is in main thread, we could optimize it too but will
have to refactor our workflow a bit). The speed went up from ~2500
(~1900 originally before previous optimizations) to ~2700 sentences/s.
With this change adding more threads than cpus don't help (actually
decreases performance) so we could probably get rid of the new
internal property.
Just to note: there's slight change in output: as we don't split the
check on each paragraph change in the output some sentences with
errors will have the beginning of the next sentence (beyond paragraph
break). Hopefully it's not a big deal.
I will need to work on cleaning thigs up, add changes for
SameRuleGroupFilter and then will create another branch for everybody
to test it out.
Andriy
2015-02-20 8:10 GMT-05:00 Daniel Naber <daniel.na...@languagetool.org>:
> On 2015-02-19 22:16, Andriy Rysin wrote:
>
>> I've merged multithreading branch into master. Please try it out when
>> you have a chance and let me know if you see any issues.
>
> Thanks. Some small cleanup ideas:
>
> -setThreadPoolSize should probably be a parameter of the constructor, as
> calling it after thread pool setup would fail anyway ("Thread pool
> already initialized")
> -Does newFixedThreadPool need to use lazy init? If it gets initialized
> in the constructor, it can also be made final.
> -It can be 'threadPool' I think, no need for the 'new' and 'fixed' in
> the variable name.
>
> Regards
> Daniel
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
> _______________________________________________
> Languagetool-devel mailing list
> Languagetool-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/languagetool-devel
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Languagetool-devel mailing list
Languagetool-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/languagetool-devel
