The tokenizer.perl default is still buffered and the new -b option disables Perl buffering. The detokenizer removes the %|++; line, which changes the script's default to buffered. The new -b option disables Perl buffering. The usage help prompt is also updated with the changes.
Maybe someone can update the repository. Thanks, TomOn Mon, 09 Jan 2012 12:59:30 +0000, Herve Saint-Amand <[email protected]> wrote:
Hi, I added that $|++; a while ago so that a script could fork offdetokenizer.perl, keep it running as a child process, feed it input lineby line and synchronously read back the corresponding output. As far as I can see it's OK to just always unbuffer output. I don't think it can break anything, but honestly I don't know what effect it has on I/O performance. For what it's worth I always add that line to every Perl script I write. I'm cool with the command-line switch too. Herve On 07/01/12 11:58, Tom Hoar wrote:I just found that detokenizer.perl has a variant of the proposal belowon lines 8 & 9.# This added by Herve Saint-Amand for compatibility with translate.cgi$|++;For consistency between the two scripts, I propose adding a command-line option to both that disables buffering if the user wishes, and a line inthe usage help. while (@ARGV) { $_ = shift; /^-b$/ && ($| = 1, next); if ($HELP) { print " -b ... disable Perl buffering.\n"; On Thu, 05 Jan 2012 20:28:01 +0700, Tom Hoar <[email protected]> wrote:The current tokenizer.perl and detokenizer.perl script buffer the print output. Adding this line to the beginning of each script: $| = 1;disables buffering and forces Perl to flush each line. Can this be added?Thanks, Tom_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
detokenizer.perl
Description: Perl program
tokenizer.perl
Description: Perl program
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
