Dear Emmanuel:
> cat phrasetable.tt | sed 's/|||/~/g' | awk -F "~" '{ print $1 "|||" $2
> "|||" $5}' > filteredtable.tt
You might find interresting to know that this works as well:
gawk -F " \\\|\\\|\\\| " '{ print $1 " ||| " $2 " ||| " $5 }'
phrasetable.tt > filteredtable.tt
(yes, awk accepts multiword field separators... and you have to escape
the pipe character three times)
Although there is few difference, this command line only uses one
single process, so in case you have a limited number of processes or
just 1CPU it might be helpful, and it should be faster as well, since
you are avoiding piping input and output.
However, I don't really understand why you say it produces some
invalid lines... maybe you could give an example?
By the way, in other cases you might be interrested in changing the
separator, if your training set is lowercased, you can replace "|||"
by e.g. "A", since in a lowercased corpus you have the certainty that
the character A (and B, C, D, ...) won't appear either.
Cheers!
Germán
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
