hi zahurul
your phrase table doesn't appear to be sorted. When i run
grep -n "^0 ( 0,0 % ) 0 " phrase-table3000
the lines with the source you meantioned
0 ( 0,0 % ) 0
appears on line 538, 540, 550, 774
the phrase table needs to be sorted before binarizing. To sort the file, run
export LC_ALL=C
sort phrase-table3000 > phrase-table3000.sorted
On 22/02/2010 09:54, Zahurul Islam wrote:
Hi,
My phrase table is very big (15 GB without compression). I was
confused to follow your instruction. I took first 3000 line from the
phrase table and tried to build binary phrase table. I met that error
again. I deleted line:386. Again I met same kind of error at line 539:
line(539): '0 ( 0,0 % ) 0 ||| 0 ( 0.0 % ) 0 ||| (0) (1) (2) (3) (4)
(5) ||| (0) (1) (2) (3) (4) (5) ||| 0.84 0.248297 0.933333 0.136567 2.718'
f: 6 13 15 5 3 7 13
Abort
I have attached the experimental portion of phrase table with this
email. I hope it will help us to find out the problem. Thank you very
much for help.
Regards,
Zahurul
On Fri, Feb 19, 2010 at 11:31 PM, Hieu Hoang <[email protected]
<mailto:[email protected]>> wrote:
can you check if there's 2 lines that have the same source? if so,
can you send me the 2 lines
On 19/02/2010 16:21, Zahurul Islam wrote:
Hi,
I was trying to generate binary phrase table using following
command: ("LC_ALL=C" was set)
$zcat phrase-table.gz | sort | /X/moses/misc/processPhraseTable
-ttable 0 0 - -nscores 5 -out phrase-table
I met following error:
ERROR: xsource phrase already inserted (B)!
line(386): '000 000 ||| 000 000 other ||| (0) (1) ||| (0) (1) ()
||| 1 0.219318 0.0194175 0.000347081 2.718'
f: 2 6 6
Abort
Any idea what's going wrong here?
Thanks in advance.
Regards,
Zahurul Islam
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
