Hi Germán,
Thanks a lot for the hints, I didn't know awk accepts multiword separator.
I couldn't give you an example of the invalid lines I encountered
since I don't have the files anymore. However, very few lines were
concerned (maybe ten out of several millions). Maybe the lines were
invalid in the first place, e.g before filtering.
Cheers, Emmanuel
On Thu, Apr 10, 2008 at 9:25 AM, Germán Sanchis Trilles
<[EMAIL PROTECTED]> wrote:
> Dear Emmanuel:
>
>
>
> > cat phrasetable.tt | sed 's/|||/~/g' | awk -F "~" '{ print $1 "|||" $2
> > "|||" $5}' > filteredtable.tt
> >
>
> You might find interresting to know that this works as well:
>
> gawk -F " \\\|\\\|\\\| " '{ print $1 " ||| " $2 " ||| " $5 }'
> phrasetable.tt > filteredtable.tt
>
> (yes, awk accepts multiword field separators... and you have to escape the
> pipe character three times)
>
> Although there is few difference, this command line only uses one single
> process, so in case you have a limited number of processes or just 1CPU it
> might be helpful, and it should be faster as well, since you are avoiding
> piping input and output.
>
> However, I don't really understand why you say it produces some invalid
> lines... maybe you could give an example?
>
> By the way, in other cases you might be interrested in changing the
> separator, if your training set is lowercased, you can replace "|||" by e.g.
> "A", since in a lowercased corpus you have the certainty that the character
> A (and B, C, D, ...) won't appear either.
>
> Cheers!
>
> Germán
>
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
