hi lang
this could arise for a number of reasons. TO solve it:
1. make sure the phrase-table.half were sorted, and make sure that you set
LC_ALL=C
when you sorted the extract files AND the phrase-table.half files.
2. double check that you don't have control characters in your corpus, and
that you haven't run out of disk space.
3. check for any odd characters in the lines in files phrase-table.half
preceeding the error line 1167912.
On Mon, Dec 26, 2011 at 7:49 AM, Bill_Lang(Gmail) <[email protected]>wrote:
> Hi moses friends,
> I am running moses hierarchical training scripts several times. When
> I used small corpus (100,000 sentence pairs), everything was ok with the
> final bleus. But when I used one bigger curpus (about 300,000 sentence
> pairs), there were the same errors as follows:
> ------------------------------------------------------
> (6.6) consolidating the two halves @ Sun Dec 25 19:39:38 SGT 2011
> Executing: mosesdecoder/scripts/training/phrase-extract/consolidate
> training/model/rule-table.half.f2e
> training/model/rule-table.half.e2f.sorted training/model/rule-table
> --Hierarchical
> Consolidate v2.0 written by Philipp Koehn
> consolidating direct and indirect rule tables
> processing hierarchical rules
> ...........ERROR: source phrase does not match in line 1167912: '[X][X]
> and the republic of korea [rok] [X]' != '[X][X] and the rok [X]'
> Exit code: 1
> ERROR: Consolidating the two phrase table halves failed at
> mosesdecoder/scripts/training/train-model.perl line 1473.
> ------------------------------------------------------
>
> I used 8 and 5 for -max-phrase-length to train the hierarchical model, but
> the error was still the same.
>
> Is there any friend ever encounter the same question? Thanks in advance.
>
> Merry Christmas.
>
> --
> Best Regards,
> Lang Jun
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
