What would be really nice is having the log lines prefixed with the thread number and/or sentence number plus some kind of locking to ensure that the lines don't mix. This would give you real-time logging, and if you want the logs separated, you could easily use sort or grep to do that. But I don't know if it's worth the effort implementing that.
/Christian On Sep 14, 2011, at 2:59 PM, Barry Haddow wrote: > Kenneth's right. The logging output was intentionally left untouched when the > multi-threading was added, and I can't see any reason to change this. If > you're debugging using verbose output, then you nearly always want to run > single-threaded. > > cheers - Barry > > On Wednesday 14 September 2011 13:39:12 Kenneth Heafield wrote: >> So what exactly is the issue? Progress can be monitored with stdout. >> If stderr is queued, then you won't get sub-sentential progress anyway. >> >> I'd rather stderr tell me what it's doing so if/when there's a segfault, >> I have a place to start. >> >> Kenneth >> >> On 09/14/11 13:32, Phil Williams wrote: >>> Yes, that would work, it just needs someone to spend the time going >>> through the source and fixing the logging code. It's a bigger -- and >>> less critical -- job than for the rest of the output, so hasn't been >>> done yet. >>> >>> Phil >>> >>> On 14 Sep, 2011,at 01:12 PM, Taylor Rose >>> >>> <[email protected]> wrote: >>>> Couldn't you queue the stderr logging to solve this issue? >>>> >>>> On Wed, 2011-09-14 at 08:55 +0000, Phil Williams wrote: >>>>> Hi Tom, >>>>> >>>>> >>>>> yes, that's right. In multithreaded moses / moses_chart, the >>>>> translations, n-best, and trace output for sentence n are all queued >>>>> until the output from sentence n-1 has been written. The queueing >>>>> doesn't happen for the logging output that goes to stderr -- it's >>>>> written immediately -- so it will appear out of order and out of sync >>>>> with the rest of the output. >>>>> >>>>> >>>>> Phil >>>>> >>>>> On 14 Sep, 2011,at 01:33 AM, Tom Hoar >>>>> <[email protected] >>>> >>>> <mailto:[email protected]>> wrote: >>>>>> Phil, >>>>>> >>>>>> Re "output to stderr will be messed up"... do you mean that the >>>>>> order and timing of stderr output will be out of synch with the >>>>>> output to stdout? I found this to be the case with multi-threaded >>>>>> moses and therefore stderr output can not be used to monitor >>>>>> progress and/or control workflows. >>>>>> >>>>>> Tom >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 13 Sep 2011 19:27:47 +0000 (GMT), Phil Williams >>>>>> >>>>>> <[email protected] <mailto:philip.williams@maccom>> wrote: >>>>>>> I think GENERAL:cores sets the maximum number of active EMS steps, >>>>>>> it doesn't change the number of threads used for decoding. You >>>>>>> need to set the decoder's -threads N option in >>>>>>> TUNING:decoder-settings and/or EVALUATION:decoder-settings. >>>>>>> >>>>>>> >>>>>>> A caveat is that the output to stderr will be messed up, though >>>>>>> that's true for multi-threaded moses as well. >>>>>>> >>>>>>> >>>>>>> Phil >>>>>>> >>>>>>> On 13 Sep, 2011,at 08:11 PM, Hieu Hoang wrote: >>>>>>>> Is it as simple as setting the >>>>>>>> [GENERAL] >>>>>>>> cores = 8 >>>>>>>> in the config file, and making sure the decoding was compiled >>>>>>>> with threads? >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Moses-support mailing list >>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>> >>>>>> _______________________________________________ >>>>>> Moses-support mailing list >>>>>> [email protected] <mailto:Moses-support@mitedu> >>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>> >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> <http://mailmanmit.edu/mailman/listinfo/moses-support> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
