Hi Admin, Inc-giza taken from : https://code.google.com/p/inc-giza-pp/ And manually compiled.
The command invoked by Moses EMS(actually train-model.perl ) is : /mnt/hd1/git/mosesdecoder/training-tools/GIZA++ -CoocurrenceFile /mnt/hd1/training/working-dir-en-es/training/giza.5/en-es.cooc -c /mnt/hd1/training/working-dir-en-es/training/prepared.5/en-es-int-train.snt -hmmdumpfrequency 5 -hmmiterations 5 -m1 5 -m2 0 -m3 0 -m4 0 -m5 0 -model1dumpfrequency 0 -model2dumpfrequency 0 -model345dumpfrequency 0 -model4smoothfactor 0.4 -nodumps 0 -nsmooth 4 -o /mnt/hd1/training/working-dir-en-es/training/giza.5/en-es -oldAlPrbs /mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.hhmm.5 -oldTrPrbs /mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.thmm.5 -onlyaldumps 1 -p0 0.999 -s /mnt/hd1/training/working-dir-en-es/training/prepared.5/es.vcb -step_k 1 -t /mnt/hd1/training/working-dir-en-es/training/prepared.5/en.vcb The baseline model used is 1. It crashes with below output on console.It is a HEAP CORRPUTION. ----------- Model1: Iteration 1 number of French (target) words = 1255 initial unifrom prob = 0.000796813 Model1: (1) TRAIN CROSS-ENTROPY 16.2367 PERPLEXITY 77219 Model1: (1) VITERBI TRAIN CROSS-ENTROPY 20.8406 PERPLEXITY 1.87773e+06 Model 1 Iteration: 1 took: 0 seconds ----------- Model1: Iteration 2 number of French (target) words = 1255 initial unifrom prob = 0.000796813 Model1: (2) TRAIN CROSS-ENTROPY 5.75346 PERPLEXITY 53.9466 Model1: (2) VITERBI TRAIN CROSS-ENTROPY 9.11781 PERPLEXITY 555.562 Model 1 Iteration: 2 took: 0 seconds ----------- Model1: Iteration 3 number of French (target) words = 1255 initial unifrom prob = 0.000796813 Model1: (3) TRAIN CROSS-ENTROPY 5.61082 PERPLEXITY 48.868 Model1: (3) VITERBI TRAIN CROSS-ENTROPY 8.86927 PERPLEXITY 467.646 Model 1 Iteration: 3 took: 0 seconds ----------- Model1: Iteration 4 number of French (target) words = 1255 initial unifrom prob = 0.000796813 Model1: (4) TRAIN CROSS-ENTROPY 5.51417 PERPLEXITY 45.7014 Model1: (4) VITERBI TRAIN CROSS-ENTROPY 8.64774 PERPLEXITY 401.078 Model 1 Iteration: 4 took: 0 seconds ----------- Model1: Iteration 5 number of French (target) words = 1255 initial unifrom prob = 0.000796813 Model1: (5) TRAIN CROSS-ENTROPY 5.44065 PERPLEXITY 43.4309 Model1: (5) VITERBI TRAIN CROSS-ENTROPY 8.44812 PERPLEXITY 349.25 Model 1 Iteration: 5 took: 0 seconds Entire Model1 Training took: 0 seconds NOTE: I am doing iterations with the HMM model! Loading HMM alignments from file. *** glibc detected *** /mnt/hd1/git/mosesdecoder/training-tools/GIZA++: malloc(): memory corruption: 0x00000000011c4470 *** ======= Backtrace: ========= [0x524d22] [0x526f4d] [0x527b7b] [0x4f0fcd] [0x4d2829] [0x4d329a] [0x4d340b] [0x495c39] [0x48bdf2] [0x42822f] [0x4294ba] [0x505e66] [0x4001e9] ======= Memory map: ======== 00400000-005d9000 r-xp 00000000 08:10 6030184 /mnt/hd1/git/mosesdecoder/training-tools/GIZA++ 007d8000-007db000 rw-p 001d8000 08:10 6030184 /mnt/hd1/git/mosesdecoder/training-tools/GIZA++ 007db000-007f2000 rw-p 00000000 00:00 0 00fc4000-01438000 rw-p 00000000 00:00 0 [heap] 7f3054000000-7f3054028000 rw-p 00000000 00:00 0 7f3054028000-7f3058000000 ---p 00000000 00:00 0 7f3059db4000-7f3059db5000 rw-p 00000000 00:00 0 7fff689b0000-7fff689c5000 rw-p 00000000 00:00 0 [stack] 7fff689ff000-7fff68a00000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted (core dumped) Also core analysis using GDB yields: Core was generated by `/mnt/hd1/git/mosesdecoder/training-tools/GIZA++ -CoocurrenceFile /mnt/hd1/train'. Program terminated with signal 6, Aborted. #0 0x0000000000547ee5 in raise () (gdb) where #0 0x0000000000547ee5 in raise () #1 0x000000000050e635 in abort () #2 0x000000000051f365 in __libc_message () #3 0x0000000000524d22 in malloc_printerr () #4 0x0000000000526f4d in _int_malloc () #5 0x0000000000527b7b in malloc () #6 0x00000000004f0fcd in operator new(unsigned long) () #7 0x00000000004d2829 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () #8 0x00000000004d329a in char* std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) () #9 0x00000000004d340b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) () #10 0x0000000000495c39 in HMMTables<int, WordClasses>::readJumps(std::basic_istream<char, std::char_traits<char> >&) () #11 0x000000000048bdf2 in hmm::load_table(char const*) () #12 0x000000000042822f in StartTraining(int&) () #13 0x00000000004294ba in main () Is it a problem with inc-Giza or could it be a problem with the libs being used. Regards, Sujay
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
