Hi,
I wish that was the problem..
sdumitrescu /usr/local/trans/tools/moses > cat ../../corpus/new/dev.0.ro
| head -3 > ../../work/roen2/test3.ro
sdumitrescu /usr/local/trans/tools/moses > cat ../../corpus/new/dev.0.en
| head -3 > ../../work/roen2/test3.en
sdumitrescu /usr/local/trans/tools/moses > cat ../../work/roen2/test3.ro
pentru cantitatile care depasesc 110 % din greutatea neta estimata nu se
acorda nici_o restituire .
Comisia este informata de_catre statele membre cu_privire_la cantitatile
si la preturile oferite de ofertanti , cel t▒rziu la orele 9 dimineata (
ora Bruxelles -ului ) ▒n ziua urmatoare datei de ▒nchidere mentionate la
articolul 16 alineatul ( 3 ) .
responsabilul este abilitat sa ▒l reprezinte pe comerciant si sa ia
deciziile necesare pentru ▒ndeplinirea sarcinilor mentionate anterior .
sdumitrescu /usr/local/trans/tools/moses > cat ../../work/roen2/test3.en
No refund shall be granted for quantities exceeding 110 % of the
estimated net mass .
the Member States shall inform the Commission of the quantities and
prices offered by tenderers , no later than 9 a.m. ( Brussels time ) on
the day following the closing date referred to in Article 16 ( 3 ) .
the officer shall be empowered to represent the operator and to take the
decisions necessary for performing the tasks specified above . 2 .
sdumitrescu /usr/local/trans/work/roen2 > ./mert_1t.sh
main::create_extractor_script() called too early to check prototype at
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl line 674.
Using SCRIPTS_ROOTDIR: /usr/local/trans/tools/moses/scripts
filtering the phrase tables... Fri Jan 27 13:58:20 EET 2012
exec:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/work/roen2/*test3.ro*
Executing:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/work/roen2/*test3.ro* > filterphrases.out 2>
filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /usr/local/trans/tools/moses/moses-cmd/src/moses -threads 16
-config filtered/moses.ini -inputtype 0 -show-weights > ./features.list
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
input-factors: 0
inputtype: 0
lmodel-file: 8 0 5
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin
mapping: 0 T 0
show-weights:
threads: 16
ttable-file: 0 0 0 5
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
ttable-limit: 20
weight-d: 0.6
weight-l: 0.5000
weight-t: 0.20 0.20 0.20 0.20 0.20
weight-w: -1
Loading lexical distortion models...have 0 models
Start loading LanguageModel
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin : [0.000] seconds
Finished loading LanguageModels : [0.000] seconds
Start loading PhraseTable
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
[0.000] seconds
filePath: /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
Finished loading phrase tables : [0.000] seconds
Start loading phrase table from
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
[0.000] seconds
Reading /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
could not close file
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 using zlib
There is something fishy going on..
Stefan
On 1/27/2012 1:36 PM, Alexandru Ceausu wrote:
Hi Stefan,
It does not work in your first experiment because the test files
contain factors, but they are not specified in moses.ini.
Defined parameters (per moses.ini or switch):
input-factors: 0
mapping: 0 T 0
The second configuration works because:
input-factors: 0 1 2 3
mapping: 0 T 0 0 G 0 0 T 1 0 G 1
Best regards,
Al. Ceausu
Alexandru Ceausu, Ph.D.
http://www.computing.dcu.ie/~aceausu/
<http://www.computing.dcu.ie/%7Eaceausu/>
On Fri, Jan 27, 2012 at 10:36 AM, Stefan Dumitrescu
<[email protected] <mailto:[email protected]>> wrote:
Hi!
#sorry for the long listing: (did bold some stuff for quick viewing):
sdumitrescu /usr/local/trans/tools/moses > df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 1.4T 211G *1.1T 17%* /
udev 5.9G 4.0K 5.9G 1% /dev
tmpfs 2.4G 212K 2.4G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 5.9G 0 5.9G 0% /run/shm
..etc., no errors here - translating and mert..ing ok
Thanks,
Stefan
On 1/27/2012 11:38 AM, Hieu Hoang wrote:
hi stefan
1. make sure you haven't run out of disk space
2. make sure every sentence in your test file ends with a new line
On Fri, Jan 27, 2012 at 4:11 PM, Stefan Dumitrescu
<[email protected]
<mailto:[email protected]>> wrote:
Hi Ken,
The phrase-table.0-0.1.1 is 0 bytes long..
I have commented out the abort() command and now it just
skips over with the same error:
sdumitrescu /usr/local/trans/work/roen2 > rm -r -f 1/mert
sdumitrescu /usr/local/trans/work/roen2 > ./mert_1t.sh
main::create_extractor_script() called too early to check
prototype at
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl
<http://mert-moses.pl> line 674.
Using SCRIPTS_ROOTDIR: /usr/local/trans/tools/moses/scripts
filtering the phrase tables... Fri Jan 27 11:03:40 EET 2012
exec:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/work/roen2/test.ro <http://test.ro>
Executing:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/work/roen2/test.ro <http://test.ro> >
filterphrases.out 2> filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /usr/local/trans/tools/moses/moses-cmd/src/moses
-threads 16 -config filtered/moses.ini -inputtype 0
-show-weights > ./features.list
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
input-factors: 0
inputtype: 0
lmodel-file: 8 0 5
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin
mapping: 0 T 0
show-weights:
threads: 16
ttable-file: 0 0 0 5
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
ttable-limit: 20
weight-d: 0.6
weight-l: 0.5000
weight-t: 0.20 0.20 0.20 0.20 0.20
weight-w: -1
Loading lexical distortion models...have 0 models
Start loading LanguageModel
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin :
[0.000] seconds
Finished loading LanguageModels : [0.000] seconds
Start loading PhraseTable
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
: [0.000] seconds
filePath:
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
Finished loading phrase tables : [0.000] seconds
Start loading phrase table from
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
: [0.000] seconds
Reading
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
could not close file
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
using zlib
Finished loading phrase tables : [0.000] seconds
MERT starting values and ranges for random generation:
d = 0.600 ( 0.00 .. 1.00)
lm = 0.500 ( 0.00 .. 1.00)
w = -1.000 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
run 1 start at Fri Jan 27 11:03:49 EET 2012
Parsing --decoder-flags: |-threads 16|
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
(1) run decoder to produce n-best lists
.. and continues to the second run when it stops normally.
I have created the test.ro/en <http://test.ro/en> files which
contain just one sentence, for testing purposes (instead of
dev.ro/en <http://dev.ro/en>).
Maybe the error is not gzip related, I'm guessing the
phrase-table.0-0.1.1 does not get written in the first place?
Stefan
p.s. the script file again:
sdumitrescu /usr/local/trans/work/roen2 > cat mert_1t.sh
#!/bin/sh
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl
<http://mert-moses.pl> \
/usr/local/trans/work/roen2/test.ro <http://test.ro> \
/usr/local/trans/work/roen2/test.en \
/usr/local/trans/tools/moses/moses-cmd/src/moses \
/usr/local/trans/work/roen2/1/model/moses.ini \
--mertdir /usr/local/trans/tools/moses/mert \
--working-dir /usr/local/trans/work/roen2/1/mert \
--rootdir /usr/local/trans/tools/moses/scripts \
--decoder-flags "-threads 16"
On 1/26/2012 4:55 PM, Kenneth Heafield wrote:
Hi,
That's my error message. Unfortunately, zlib doesn't provide a
nice
way to extract a more verbose error from gzclose:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=354310
It's possible that the MERT filtering didn't work/complete
properly,
leaving a corrupt gzipped file. Can you verify its integrity with
zcat /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
>/dev/null
for me? Also, based on the number of asterisks in your output, it looks
like an exception was thrown somewhere early in the file, processing
this exception led this destructor being called, and the true exception
was masked because the destructor won the race to kill the process. Try
commenting out abort() in util/file_piece.cc line 61. That will let the
process continue, possibly giving you the exception being thrown as a
more detailed error message.
Kenneth
On 01/26/2012 07:27 AM, Stefan Dumitrescu wrote:
Hello again,
I have a strange problem with MERT. I have trained several models and I
am attempting to tune them. While for the factored models the MERT
scripts run ok, for the unfactored model it fails with this error:
sdumitrescu /usr/local/trans/work/roen2> ./mert_1.sh
main::create_extractor_script() called too early to check prototype at
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl
<http://mert-moses.pl> line 674.
Using SCRIPTS_ROOTDIR: /usr/local/trans/tools/moses/scripts
filtering the phrase tables... Thu Jan 26 14:14:07 EET 2012
exec:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/corpus/new/dev.0.ro <http://dev.0.ro>
Executing:
/usr/local/trans/tools/moses/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/usr/local/trans/work/roen2/1/model/moses.ini
/usr/local/trans/corpus/new/dev.0.ro <http://dev.0.ro> >
filterphrases.out 2>
filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /usr/local/trans/tools/moses/moses-cmd/src/moses -threads 16
-config filtered/moses.ini -inputtype 0 -show-weights> ./features.list
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
input-factors: 0
inputtype: 0
lmodel-file: 8 0 5
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin
mapping: 0 T 0
show-weights:
threads: 16
ttable-file: 0 0 0 5
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
ttable-limit: 20
weight-d: 0.6
weight-l: 0.5000
weight-t: 0.20 0.20 0.20 0.20 0.20
weight-w: -1
Loading lexical distortion models...have 0 models
Start loading LanguageModel
/usr/local/trans/corpus/lm/big/big.5g.surface.en.lm.bin : [0.000]
seconds
Finished loading LanguageModels : [0.000] seconds
Start loading PhraseTable
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
[0.000] seconds
filePath:
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
Finished loading phrase tables : [0.000] seconds
Start loading phrase table from
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 :
[0.000] seconds
Reading /usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
*could not close file
/usr/local/trans/work/roen2/1/mert/filtered/phrase-table.0-0.1.1 using
zlib*
Aborted
Exit code: 134
Failed to run moses with the config filtered/moses.ini at
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl
<http://mert-moses.pl> line 1072.
The script for model 1 (mert_1.sh) is:
/usr/local/trans/tools/moses/scripts/training/mert-moses.pl
<http://mert-moses.pl> \
/usr/local/trans/corpus/new/dev.ro <http://dev.ro> \
/usr/local/trans/corpus/new/dev.en \
/usr/local/trans/tools/moses/moses-cmd/src/moses \
/usr/local/trans/work/roen2/1/model/moses.ini \
--mertdir /usr/local/trans/tools/moses/mert \
--working-dir /usr/local/trans/work/roen2/1/mert \
--rootdir /usr/local/trans/tools/moses/scripts \
--decoder-flags "-threads 16"
and it is similar to the other scripts (instead of folder 1 i have
folder 2,3, etc) for which i get no error and tuning is working ok.
Also, all the models are ok, i have used them to translate without
problems.
Any idea why i get this error?
Thank you,
Stefan
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
