Heyhey On Thu, Jan 28, 2016 at 08:57:49AM -0500, Andriy Rysin wrote: > yes, that's the case I tried with # of callables = # of rules (see my > previous email), the wait time went down quite a bit (as expected) but > the overall processing time went up, I suspect because of split/merge > overhead. But this depends heavily of type/number of rules, the text > and cpu (e.g. if rule processing time is more unbabalanced that in > Ukrainian case then increasing # of callables will help, otherwise the > effect could be reverse) so we would have to try other languages with
I may have missed a statistic in your earlier mail but wouldn't splitting up the text in sentences and then sending batches of them to different threads result in the most evenly load distribution? Because of the result merging overhead this would only make sense when the number of lines crosses a certain threshhold. Cheers, Silvan > different number of callables to see what's the best approach. > I know we have regular wikipedia checks for some languages - that > could be a good benchmarking test. > > Regards, > Andriy > > 2016-01-28 8:47 GMT-05:00 Dominique Pellé <dominique.pe...@gmail.com>: > > Andriy Rysin wrote: > > > > > >> Then I realized that in the check method we split rules into callables > >> and their count is # of cores available (in my case 8), as I have 347 > >> rules this means each bucket is 43 rules and rules being not equal in > >> complexity this could lead to quite unequal time for each thread. > > > > Hi Andriy > > > > Thanks for having a look at multi-thread performances. > > I don't know the code as much as you do. But if we indeed > > split the number of rules equally before processing them, then > > it seems bad for balancing the work. > > > > Can't we instead have a queue with all rules to be processed? > > When a thread is ready to do work, it picks the next rule to process > > from the queue. So load would be well balanced, even if some rules > > are 10x slower than others. With such a queue, a thread that picks > > up an expensive rules would end up processing less rules than a > > thread that picks up fast rules, keeping all CPUs busy, as much > > as possible. > > > > Regards > > Dominique > > > > ------------------------------------------------------------------------------ > > Site24x7 APM Insight: Get Deep Visibility into Application Performance > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > > Monitor end-to-end web transactions and take corrective actions now > > Troubleshoot faster and improve end-user experience. Signup Now! > > http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 > > _______________________________________________ > > Languagetool-devel mailing list > > Languagetool-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/languagetool-devel > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 > _______________________________________________ > Languagetool-devel mailing list > Languagetool-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/languagetool-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 _______________________________________________ Languagetool-devel mailing list Languagetool-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/languagetool-devel
