Dear Qin, I don't think so because right before my linux crashes, I was using this corpus and all the things were right. Could it be from the OS and its settings? my previous OS was debian and now I installed kubuntu.
On Tue, Dec 28, 2010 at 9:04 PM, Qin Gao <[email protected]> wrote: > Seems th giza crashes when or after loading the corpus, you should check > the corpus, whether there are empty lines, that is a common cause of the > problem. > --Q > > > > On Tue, Dec 28, 2010 at 12:31 PM, amin farajian <[email protected]>wrote: > >> snt2cooc.out is installed, and it can be seen by moses. I reinstalled >> giza++ with and without the flag and I tested moses in both situations, but >> nothing changed. >> Here are the messages that appeared on the screen: >> >> executing >> /home/amin/mt/work/baseline/phraseBased/3lm/steps/1/TRAINING_run-giza-inverse.1 >> via sh (1) >> executing >> /home/amin/mt/work/baseline/phraseBased/3lm/steps/1/TRAINING_run-giza.1 via >> sh (2) >> *** buffer overflow detected ***: /home/amin/mt/bin/GIZA++ terminated >> ======= Backtrace: ========= >> [0x817af29] >> [0x817aedd] >> [0x817acf8] >> [0x815cd3e] >> [0x8187eb5] >> [0x817ad94] >> [0x817aced] >> [0x806ce9d] >> [0x806d903] >> [0x8073b70] >> [0x8147b3f] >> [0x8048201] >> ======= Memory map: ======== >> 00fa7000-00fa8000 r-xp 00000000 00:00 0 [vdso] >> 08048000-081fb000 r-xp 00000000 08:04 804586 /home/amin/mt/bin/GIZA++ >> 081fc000-081fe000 rw-p 001b3000 08:04 804586 /home/amin/mt/bin/GIZA++ >> 081fe000-08206000 rw-p 00000000 00:00 0 >> 08d39000-08dad000 rw-p 00000000 00:00 0 [heap] >> bfd80000-bfda1000 rw-p 00000000 00:00 0 [stack] >> *** buffer overflow detected ***: /home/amin/mt/bin/GIZA++ terminated >> ======= Backtrace: ========= >> [0x817af29] >> [0x817aedd] >> [0x817acf8] >> [0x815cd3e] >> [0x8187eb5] >> [0x817ad94] >> [0x817aced] >> [0x806ce9d] >> [0x806d903] >> [0x8073b70] >> [0x8147b3f] >> [0x8048201] >> ======= Memory map: ======== >> 00cc2000-00cc3000 r-xp 00000000 00:00 0 [vdso] >> 08048000-081fb000 r-xp 00000000 08:04 804586 /home/amin/mt/bin/GIZA++ >> 081fc000-081fe000 rw-p 001b3000 08:04 804586 /home/amin/mt/bin/GIZA++ >> 081fe000-08206000 rw-p 00000000 00:00 0 >> 09df6000-09e6a000 rw-p 00000000 00:00 0 [heap] >> bfaf0000-bfb11000 rw-p 00000000 00:00 0 [stack] >> step TRAINING:run-giza-inverse crashed >> step TRAINING:run-giza crashed >> number of steps doable or running: 0 >> >> and here is the content of "TRAINING_run-giza.1.STDERR": >> >> Using SCRIPTS_ROOTDIR: /home/amin/mt/bin/scripts >> Using single-thread GIZA >> (2) running giza on 5 cooc parts @ Tue Dec 28 20:35:28 IRST 2010 >> Executing: mkdir -p /home/amin/mt/work/baseline/ >> phraseBased/3lm/training/prepared.1/part1 >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part2 >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part3 >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part4 >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part5 >> (2.1a) running snt2cooc fr-en @ Tue Dec 28 20:35:28 IRST 2010 >> >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part1 >> Executing: /home/amin/mt/bin/snt2cooc.out >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part1/fr-en-int-train.snt >> > >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part1/fr-en.cooc >> END. >> (2.1a) running snt2cooc fr-en @ Tue Dec 28 20:35:28 IRST 2010 >> >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part2 >> Executing: /home/amin/mt/bin/snt2cooc.out >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part2/fr-en-int-train.snt >> > >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part2/fr-en.cooc >> END. >> (2.1a) running snt2cooc fr-en @ Tue Dec 28 20:35:28 IRST 2010 >> >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part3 >> Executing: /home/amin/mt/bin/snt2cooc.out >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part3/fr-en-int-train.snt >> > >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part3/fr-en.cooc >> END. >> (2.1a) running snt2cooc fr-en @ Tue Dec 28 20:35:28 IRST 2010 >> >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part4 >> Executing: /home/amin/mt/bin/snt2cooc.out >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part4/fr-en-int-train.snt >> > >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part4/fr-en.cooc >> END. >> (2.1a) running snt2cooc fr-en @ Tue Dec 28 20:35:28 IRST 2010 >> >> Executing: mkdir -p >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part5 >> Executing: /home/amin/mt/bin/snt2cooc.out >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/part5/fr-en-int-train.snt >> > >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/part5/fr-en.cooc >> END. >> (2.1b) running giza fr-en @ Tue Dec 28 20:35:29 IRST 2010 >> /home/amin/mt/bin/GIZA++ -CoocurrenceFile >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en.cooc -c >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr-en-int-train.snt >> -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 >> -nodumps 1 -nsmooth 4 -o >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en >> -onlyaldumps 1 -p0 0.999 -s >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb -t >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> Executing: /home/amin/mt/bin/GIZA++ -CoocurrenceFile >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en.cooc -c >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr-en-int-train.snt >> -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 >> -nodumps 1 -nsmooth 4 -o >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en >> -onlyaldumps 1 -p0 0.999 -s >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb -t >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> ERROR: Execution of: /home/amin/mt/bin/GIZA++ -CoocurrenceFile >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en.cooc -c >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr-en-int-train.snt >> -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 >> -nodumps 1 -nsmooth 4 -o >> /home/amin/mt/work/baseline/phraseBased/3lm/training/giza.1/fr-en >> -onlyaldumps 1 -p0 0.999 -s >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/en.vcb -t >> /home/amin/mt/work/baseline/phraseBased/3lm/training/prepared.1/fr.vcb >> died with signal 6, without coredump >> >> >> On Tue, Dec 28, 2010 at 8:25 PM, Qin Gao <[email protected]> wrote: >> >>> If you comment out -DBINARY_SEARCH_FOR_TTABLE then giza will use hash >>> table instead of binary search for lexicon translation model, that will use >>> more memory. The problem seems to be snt2cooc.out is not installed. My >>> suggestion is to keep the compiler flag and install snt2cooc.out, make sure >>> that can be seen by moses. It is included in both giza++ and mgiza package. >>> >>> --Q >>> >>> >>> On Tue, Dec 28, 2010 at 11:45 AM, amin farajian >>> <[email protected]>wrote: >>> >>>> Hi All, >>>> One month ago, when I wanted to use moses to build a baseline SMT, I >>>> faced a problem during training the models. moses crashed on "run-giza" and >>>> "run-giza inverse". After some searches I found that I have to uncomment a >>>> line in giza++ which contains "-DBINARY_SEARCH_FOR_TTABLE", and make it >>>> again. that time it worked. but, 2 days ago my linux crashed and it made me >>>> to reinstall it. after installing the newest version of kubuntu, I >>>> installed >>>> giza++, mkcls, srilm and moses. when I wanted to build the base-line MT, >>>> moses crashed in "run-giza" and "run-giza inverse",again. I uncommented >>>> that >>>> line in the make file of giza++ but nothing changed in running moses. What >>>> can be the cause of this problem? >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> >>>> >>> >> >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
