Hi Stefan,
It does not work in your first experiment because the test files contain
factors, but they are not specified in moses.ini.
Defined parameters (per moses.ini or switch):
input-factors: 0
mapping: 0 T 0
The second configuration works because:
input-factors: 0 1 2 3
mapping: 0 T 0 0 G 0 0 T 1 0 G 1
Best regards,
Al. Ceausu
Alexandru Ceausu, Ph.D.
http://www.computing.dcu.ie/~aceausu/
On Fri, Jan 27, 2012 at 10:36 AM, Stefan Dumitrescu <
[email protected]> wrote:
> Hi!
>
> #sorry for the long listing: (did bold some stuff for quick viewing):
>
>
> sdumitrescu /usr/local/trans/tools/moses > df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 1.4T 211G *1.1T 17%* /
> udev 5.9G 4.0K 5.9G 1% /dev
> tmpfs 2.4G 212K 2.4G 1% /run
> none 5.0M 0 5.0M 0% /run/lock
> none 5.9G 0 5.9G 0% /run/shm
>
>
>
> ..etc., no errors here - translating and mert..ing ok
>
> Thanks,
> Stefan
>
>
> On 1/27/2012 11:38 AM, Hieu Hoang wrote:
>
> hi stefan
>
> 1. make sure you haven't run out of disk space
> 2. make sure every sentence in your test file ends with a new line
>
> On Fri, Jan 27, 2012 at 4:11 PM, Stefan Dumitrescu <
> [email protected]> wrote:
>
>> Hi Ken,
>>
>> The phrase-table.0-0.1.1 is 0 bytes long..
>>
>> I have commented out the abort() command and now it just skips over with
>> the same error:
>>
>> sdumitrescu /usr/local/trans/work/roen2 > rm -r -f 1/mert
>> sdumitrescu /usr/local/trans/work/roen2 > ./mert_1t.sh
>>
>> main::create_extractor_script() called too early to check prototype at
>> /usr/local/trans/tools/moses/scripts/training/mert-moses.pl line 674.
>> Using SCRIPTS_ROOTDIR: /usr/local/trans/tools/moses/scripts
>> filtering the phrase tables... Fri Jan 27 11:03:40 EET 2012
>> exec: /usr/local/trans/tools/moses/scripts/training/
>> filter-model-given-input.pl ./filtered
>> /usr/local/trans/work/roen2/1/model/moses.ini /usr/local/trans/work/roen2/
>> test.ro
>> Executing: /usr/local/trans/tools/moses/scripts/training/
>> filter-model-given-input.pl ./filtered
>> /usr/local/trans/work/roen2/1/model/moses.ini /usr/local/trans/work/roen2/
>> test.ro > filterphrases.out 2> filterphrases.err
>>
>> Asking moses for feature names and values from filtered/moses.ini
>> Executing: /usr/local/trans/tools/moses/moses-cmd/src/moses -threads 16
>> -config filtered/moses.ini -inputtype 0 -show-weights > ./features.list
>> Defined parameters (per moses.ini or switch):
>> config: filtered/moses.ini
>> distortion-limit: 6
>> input-factors: 0
>> inputtype: 0
>> lmodel-file: 8 0 5
>> /usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin
>> mapping: 0 T 0
>> show-weights:
>> threads: 16
>> ttable-file: 0 0 0 5
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> ttable-limit: 20
>> weight-d: 0.6
>> weight-l: 0.5000
>> weight-t: 0.20 0.20 0.20 0.20 0.20
>> weight-w: -1
>> Loading lexical distortion models...have 0 models
>> Start loading LanguageModel
>> /usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin : [0.000] seconds
>> Finished loading LanguageModels : [0.000] seconds
>> Start loading PhraseTable
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 : [0.000]
>> seconds
>> filePath: /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> Finished loading phrase tables : [0.000] seconds
>> Start loading phrase table from
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 : [0.000]
>> seconds
>> Reading /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>>
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> could not close file
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 using zlib
>> Finished loading phrase tables : [0.000] seconds
>> MERT starting values and ranges for random generation:
>> d = 0.600 ( 0.00 .. 1.00)
>> lm = 0.500 ( 0.00 .. 1.00)
>> w = -1.000 ( 0.00 .. 1.00)
>> tm = 0.200 ( 0.00 .. 1.00)
>> tm = 0.200 ( 0.00 .. 1.00)
>> tm = 0.200 ( 0.00 .. 1.00)
>> tm = 0.200 ( 0.00 .. 1.00)
>> tm = 0.200 ( 0.00 .. 1.00)
>> run 1 start at Fri Jan 27 11:03:49 EET 2012
>> Parsing --decoder-flags: |-threads 16|
>> Saving new config to: ./run1.moses.ini
>> Saved: ./run1.moses.ini
>> (1) run decoder to produce n-best lists
>>
>> .. and continues to the second run when it stops normally.
>>
>> I have created the test.ro/en files which contain just one sentence, for
>> testing purposes (instead of dev.ro/en).
>>
>> Maybe the error is not gzip related, I'm guessing the
>> phrase-table.0-0.1.1 does not get written in the first place?
>>
>> Stefan
>>
>> p.s. the script file again:
>> sdumitrescu /usr/local/trans/work/roen2 > cat mert_1t.sh
>> #!/bin/sh
>>
>> /usr/local/trans/tools/moses/scripts/training/mert-moses.pl \
>> /usr/local/trans/work/roen2/test.ro \
>> /usr/local/trans/work/roen2/test.en \
>>
>> /usr/local/trans/tools/moses/moses-cmd/src/moses \
>> /usr/local/trans/work/roen2/1/model/moses.ini \
>> --mertdir /usr/local/trans/tools/moses/mert \
>> --working-dir /usr/local/trans/work/roen2/1/mert \
>> --rootdir /usr/local/trans/tools/moses/scripts \
>> --decoder-flags "-threads 16"
>>
>>
>>
>>
>>
>>
>> On 1/26/2012 4:55 PM, Kenneth Heafield wrote:
>>
>> Hi,
>>
>> That's my error message. Unfortunately, zlib doesn't provide a nice
>> way to extract a more verbose error from gzclose:
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=354310
>>
>> It's possible that the MERT filtering didn't work/complete properly,
>> leaving a corrupt gzipped file. Can you verify its integrity with
>>
>> zcat /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> >/dev/null
>>
>> for me? Also, based on the number of asterisks in your output, it looks
>> like an exception was thrown somewhere early in the file, processing
>> this exception led this destructor being called, and the true exception
>> was masked because the destructor won the race to kill the process. Try
>> commenting out abort() in util/file_piece.cc line 61. That will let the
>> process continue, possibly giving you the exception being thrown as a
>> more detailed error message.
>>
>> Kenneth
>>
>> On 01/26/2012 07:27 AM, Stefan Dumitrescu wrote:
>>
>> Hello again,
>>
>> I have a strange problem with MERT. I have trained several models and I
>> am attempting to tune them. While for the factored models the MERT
>> scripts run ok, for the unfactored model it fails with this error:
>>
>> sdumitrescu /usr/local/trans/work/roen2 > ./mert_1.sh
>> main::create_extractor_script() called too early to check prototype at
>> /usr/local/trans/tools/moses/scripts/training/mert-moses.pl line 674.
>> Using SCRIPTS_ROOTDIR: /usr/local/trans/tools/moses/scripts
>> filtering the phrase tables... Thu Jan 26 14:14:07 EET 2012
>> exec:
>> /usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
>> ./filtered
>> /usr/local/trans/work/roen2/1/model/moses.ini
>> /usr/local/trans/corpus/new/dev.0.ro
>> Executing:
>> /usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
>> ./filtered
>> /usr/local/trans/work/roen2/1/model/moses.ini
>> /usr/local/trans/corpus/new/dev.0.ro > filterphrases.out 2>
>> filterphrases.err
>> Asking moses for feature names and values from filtered/moses.ini
>> Executing: /usr/local/trans/tools/moses/moses-cmd/src/moses -threads 16
>> -config filtered/moses.ini -inputtype 0 -show-weights > ./features.list
>> Defined parameters (per moses.ini or switch):
>> config: filtered/moses.ini
>> distortion-limit: 6
>> input-factors: 0
>> inputtype: 0
>> lmodel-file: 8 0 5 /usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin
>> mapping: 0 T 0
>> show-weights:
>> threads: 16
>> ttable-file: 0 0 0 5
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> ttable-limit: 20
>> weight-d: 0.6
>> weight-l: 0.5000
>> weight-t: 0.20 0.20 0.20 0.20 0.20
>> weight-w: -1
>> Loading lexical distortion models...have 0 models
>> Start loading LanguageModel
>> /usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin : [0.000] seconds
>> Finished loading LanguageModels : [0.000] seconds
>> Start loading PhraseTable
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
>> [0.000] seconds
>> filePath: /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> Finished loading phrase tables : [0.000] seconds
>> Start loading phrase table from
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
>> [0.000] seconds
>> Reading /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> *could not close file
>> /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 using zlib*
>> Aborted
>> Exit code: 134
>> Failed to run moses with the config filtered/moses.ini at
>> /usr/local/trans/tools/moses/scripts/training/mert-moses.pl line 1072.
>>
>> The script for model 1 (mert_1.sh) is:
>> /usr/local/trans/tools/moses/scripts/training/mert-moses.pl \
>> /usr/local/trans/corpus/new/dev.ro \
>> /usr/local/trans/corpus/new/dev.en \
>> /usr/local/trans/tools/moses/moses-cmd/src/moses \
>> /usr/local/trans/work/roen2/1/model/moses.ini \
>> --mertdir /usr/local/trans/tools/moses/mert \
>> --working-dir /usr/local/trans/work/roen2/1/mert \
>> --rootdir /usr/local/trans/tools/moses/scripts \
>> --decoder-flags "-threads 16"
>>
>> and it is similar to the other scripts (instead of folder 1 i have
>> folder 2,3, etc) for which i get no error and tuning is working ok.
>> Also, all the models are ok, i have used them to translate without
>> problems.
>>
>> Any idea why i get this error?
>>
>> Thank you,
>> Stefan
>>
>>
>> _______________________________________________
>> Moses-support mailing
>> [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing
>> [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> [email protected]
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
