Hi Hwidong
Ah, went through your model and it segfaults like you said. There is a
bug when using the word 'X'.
I've checked in the fix
http://mosesdecoder.svn.sourceforge.net/viewvc/mosesdecoder?view=rev&revision=2643
Thx for phrase table
Hieu
Hwidong Na wrote:
> Hi Hieu,
>
> Here is the ini file (run1.moses.ini) and a fuller trace of output
> (log). The rule table used in the decoding is not much large so that you
> can download it. (http://home.postech.ac.kr/~leona/rule-table.gz) It
> will help you reproduce the error.
>
> It is also strange for me since I do not suffer from the X problem with
> the sample data (http://www.statmt.org/moses/download/sample-models.tgz)
>
> Best regards,
> 2009-12-30 (수), 09:18 +0000, Hieu Hoang:
>
>> the X problem is strange. I can't reproduce the error, eg. for input
>> X the man bought the house
>> or with an X in the phrase table
>> [X] [X] ||| X ||| hello ||| ||| 0.17
>>
>> can you send me the ini file and a fuller trace of the output?
>>
>> thx
>> hieu
>>
>> Hwidong Na wrote:
>>
>>> Hi all.
>>>
>>> I mean the decoder "moses_chart" and the script "mert-moses-new.pl" in
>>> the mt3_chart branch. (I follow "Compiling Chart Decoder" in
>>> http://www.statmt.org/moses/?n=Development.GetStarted)
>>>
>>> Thanks to John, I try the option "--no-filter-phrase-table" and it works
>>> in a small size corpus. It seems that there is no difference between
>>> "mert-moses.pl" and "mert-moses-new.pl" with the option, i.e.
>>> "mert-moses.pl" also supports the option.
>>>
>>> There is an another problem during the tree-based decoding. If a source
>>> sentence has a word "X", then "moses_chart" gives a segmentation fault
>>> as following.
>>>
>>> Translating: <s> X 1 の 好ましい 範囲 は 、 15 〜 30 ％ まで の 深 さ で ある 。 </s>
>>> ||| [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1)
>>> ...ellipsis...
>>> [0..0]=1 [1..1]=2 [2..2]=50 [3..3]=200 ...ellipsis... [0..1]=2
>>> [1..2]=Segmentation fault
>>> Exit code: 139
>>> The decoder died. CONFIG WAS -w 0.000000 -lm 0.500000 -d 0.500000
>>>
>>> It doesn't when I replace "X" with another word such as "-X-".
>>>
>>> 2009-12-29 (화), 21:00 +0000, Hieu Hoang:
>>>
>>>
>>>> You mean the decoder in the mt3_chart branch?
>>>>
>>>> I'm not sure the
>>>> mert-moses-new.pl
>>>> works yet. I think only the
>>>> mert-moses.pl
>>>> is working.
>>>>
>>>> also, make sure you're using the script in the mt_chart branch,
>>>> there's niggly small differences.
>>>>
>>>> 2009/12/29 John Morgan <[email protected]>
>>>> There is an option --no-filter-phrase-table for
>>>> mert-moses-new.pl.
>>>>
>>>>
>>>>
>>>> On 12/29/09, Hwidong Na <[email protected]> wrote:
>>>> > I also have a problem with tuning a tree-based system.
>>>> >
>>>> > I use a parallel text without syntactic annotation to train
>>>> the
>>>> > tree-based system. Then the extracted rule-table has the
>>>> only
>>>> > non-terminal on both source and target side. (in
>>>> Japanese-to-English)
>>>> >
>>>> > $ gzip -dc <path-to-model>/rule-table.gz | sort -k8
>>>> -t'|' -nr | head
>>>> > -n5
>>>> > [X] [X] ||| 高 開口 数 による 高 解像度 の ||| with
>>>> high resolution by means of
>>>> > a high numerical aperture ||| ||| 1 2.07446e-05 1
>>>> 2.2417e-08 2.718 |||
>>>> > 0.0232558 0.0232558
>>>> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high
>>>> [X] high numerical
>>>> > aperture ||| 5-2 ||| 1 0.000225055 0.5 0.000282326 2.718 |||
>>>> 0.0232558
>>>> > 0.0465116
>>>> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high
>>>> [X] a high numerical
>>>> > aperture ||| 5-2 ||| 1 0.000225055 0.5 1.64576e-05 2.718 |||
>>>> 0.0232558
>>>> > 0.0465116
>>>> > [X] [X] ||| 高 開口 数 [X] 解像度 の ||| [X]
>>>> resolution by means of a high
>>>> > numerical aperture ||| 3-0 ||| 1 0.00589926 1 5.15592e-07
>>>> 2.718 |||
>>>> > 0.0232558 0.0232558
>>>> > [X] [X] ||| 高 開口 数 [X] の ||| [X] of a high
>>>> numerical aperture |||
>>>> > 3-0 ||| 1 0.0503172 1 0.000170465 2.718 ||| 0.0232558
>>>> 0.0232558
>>>> >
>>>> >
>>>> > Then passing "moses_chart" to "mert-moses-new" gives
>>>> something wrong.
>>>> > The first step of decoding is filtering of
>>>> translation-table. The output
>>>> > message of filtering scripts shows
>>>> >
>>>> > filtering <path-to-model>/rule-table.gz ->
>>>> > <path-to-tuning>/filtered/phrase-table.0-0.1...
>>>> > 0 of 550276 phrases pairs used (0.00%) - note: max
>>>> length 10
>>>> > filtering <path-to-model>/glue-grammar ->
>>>> > <path-to-tuning>/filtered/phrase-table.0-0.2...
>>>> > 0 of 3 phrases pairs used (0.00%) - note: max length
>>>> 10
>>>> >
>>>> > It seems wrong. So how can I filter correctly? (or just do
>>>> not filter?)
>>>> >
>>>> >> From: Philipp Koehn <pko...@...>
>>>> >> Subject: Re: tuning tree-based models
>>>> >> Newsgroups: gmane.comp.nlp.moses.user
>>>> >> Date: 2009-12-28 23:56:11 GMT (12 hours and 2 minutes ago)
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> this should work... Does the moses process generate a
>>>> proper n-best list
>>>> >> file?
>>>> >> There may be something wrong with running the decoder.
>>>> >>
>>>> >> Regarding the section "non-terminals" in the moses.ini file
>>>> - don't worry,
>>>> >> this
>>>> >> is just a list of special non-terminals that are used for
>>>> unknown words
>>>> >> etc.
>>>> >>
>>>> >> -phi
>>>> >>
>>>> >> On Mon, Dec 28, 2009 at 2:37 PM, John Morgan
>>>> >> <johnjosephmor...@...> wrote:
>>>> >> > Hello All,
>>>> >> > I want to tune a tree-based trained system. I pass the
>>>> moses_chart
>>>> >> > decoder to the mert-moses-new.perl script. The script
>>>> seems to make 1
>>>> >> > decoding run producing features.dat and scores.dat files,
>>>> but it
>>>> >> > writes an empty extract.out file. Then the script stops.
>>>> >> > I also noticed that the train-moses.perl script only
>>>> writes out a "X"
>>>> >> > under [non-terminals] in the moses.ini file. I have
>>>> several
>>>> >> > non-terminals on the input side.
>>>> >> > Thanks,
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Regards,
>>>> >> > John J Morgan
>>>> >> > _______________________________________________
>>>> >> > Moses-support mailing list
>>>> >> > moses-supp...@...
>>>> >> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> >> >
>>>> > --
>>>> > Hwidong Na <[email protected]>
>>>> > POSTECH
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Moses-support mailing list
>>>> > [email protected]
>>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> John J Morgan
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>>
>>>>
>>>> [email protected]
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
