HI Hieu-Hong,
Thank you for your reply.
We had used the baseline model created by MGiza , and then used
inc-giza for incremental training.
So probably the file formats were incompatible.Hence inc-giza
crashed.
But we are still struggling to do Incremental training using
MGiza.
There is a separate post for the same.
Please let us know on the same.
Thanks and Regards,
Sujay
From: Hieu Hoang [mailto:[email protected]]
Sent: Monday, May 25, 2015 10:23 AM
To: Hegde, Sujay; [email protected]
Cc: Venkatapathy, Sriram (Calendar); MudaliarMudaliar, Preeti J
Subject: Re: [Moses-support] Incremental Giza(Inc-giza) crashes with core dump
when doing Incremental training using MOSES EMS
seems like no-one knows. If you managed to figure it out, please tell us
On 20/05/2015 14:32, Hegde, Sujay wrote:
Hi Admin,
Inc-giza taken from : https://code.google.com/p/inc-giza-pp/
And manually compiled.
The command invoked by Moses EMS(actually train-model.perl ) is :
/mnt/hd1/git/mosesdecoder/training-tools/GIZA++ -CoocurrenceFile
/mnt/hd1/training/working-dir-en-es/training/giza.5/en-es.cooc -c
/mnt/hd1/training/working-dir-en-es/training/prepared.5/en-es-int-train.snt
-hmmdumpfrequency 5 -hmmiterations 5 -m1 5 -m2 0 -m3 0 -m4 0 -m5 0
-model1dumpfrequency 0 -model2dumpfrequency 0 -model345dumpfrequency 0
-model4smoothfactor 0.4 -nodumps 0 -nsmooth 4 -o
/mnt/hd1/training/working-dir-en-es/training/giza.5/en-es -oldAlPrbs
/mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.hhmm.5 -oldTrPrbs
/mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.thmm.5 -onlyaldumps 1
-p0 0.999 -s /mnt/hd1/training/working-dir-en-es/training/prepared.5/es.vcb
-step_k 1 -t /mnt/hd1/training/working-dir-en-es/training/prepared.5/en.vcb
The baseline model used is 1.
It crashes with below output on console.It is a HEAP CORRPUTION.
-----------
Model1: Iteration 1
number of French (target) words = 1255
initial unifrom prob = 0.000796813
Model1: (1) TRAIN CROSS-ENTROPY 16.2367 PERPLEXITY 77219
Model1: (1) VITERBI TRAIN CROSS-ENTROPY 20.8406 PERPLEXITY 1.87773e+06
Model 1 Iteration: 1 took: 0 seconds
-----------
Model1: Iteration 2
number of French (target) words = 1255
initial unifrom prob = 0.000796813
Model1: (2) TRAIN CROSS-ENTROPY 5.75346 PERPLEXITY 53.9466
Model1: (2) VITERBI TRAIN CROSS-ENTROPY 9.11781 PERPLEXITY 555.562
Model 1 Iteration: 2 took: 0 seconds
-----------
Model1: Iteration 3
number of French (target) words = 1255
initial unifrom prob = 0.000796813
Model1: (3) TRAIN CROSS-ENTROPY 5.61082 PERPLEXITY 48.868
Model1: (3) VITERBI TRAIN CROSS-ENTROPY 8.86927 PERPLEXITY 467.646
Model 1 Iteration: 3 took: 0 seconds
-----------
Model1: Iteration 4
number of French (target) words = 1255
initial unifrom prob = 0.000796813
Model1: (4) TRAIN CROSS-ENTROPY 5.51417 PERPLEXITY 45.7014
Model1: (4) VITERBI TRAIN CROSS-ENTROPY 8.64774 PERPLEXITY 401.078
Model 1 Iteration: 4 took: 0 seconds
-----------
Model1: Iteration 5
number of French (target) words = 1255
initial unifrom prob = 0.000796813
Model1: (5) TRAIN CROSS-ENTROPY 5.44065 PERPLEXITY 43.4309
Model1: (5) VITERBI TRAIN CROSS-ENTROPY 8.44812 PERPLEXITY 349.25
Model 1 Iteration: 5 took: 0 seconds
Entire Model1 Training took: 0 seconds
NOTE: I am doing iterations with the HMM model!
Loading HMM alignments from file.
*** glibc detected *** /mnt/hd1/git/mosesdecoder/training-tools/GIZA++:
malloc(): memory corruption: 0x00000000011c4470 ***
======= Backtrace: =========
[0x524d22]
[0x526f4d]
[0x527b7b]
[0x4f0fcd]
[0x4d2829]
[0x4d329a]
[0x4d340b]
[0x495c39]
[0x48bdf2]
[0x42822f]
[0x4294ba]
[0x505e66]
[0x4001e9]
======= Memory map: ========
00400000-005d9000 r-xp 00000000 08:10 6030184
/mnt/hd1/git/mosesdecoder/training-tools/GIZA++
007d8000-007db000 rw-p 001d8000 08:10 6030184
/mnt/hd1/git/mosesdecoder/training-tools/GIZA++
007db000-007f2000 rw-p 00000000 00:00 0
00fc4000-01438000 rw-p 00000000 00:00 0 [heap]
7f3054000000-7f3054028000 rw-p 00000000 00:00 0
7f3054028000-7f3058000000 ---p 00000000 00:00 0
7f3059db4000-7f3059db5000 rw-p 00000000 00:00 0
7fff689b0000-7fff689c5000 rw-p 00000000 00:00 0 [stack]
7fff689ff000-7fff68a00000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Aborted (core dumped)
Also core analysis using GDB yields:
Core was generated by `/mnt/hd1/git/mosesdecoder/training-tools/GIZA++
-CoocurrenceFile /mnt/hd1/train'.
Program terminated with signal 6, Aborted.
#0 0x0000000000547ee5 in raise ()
(gdb) where
#0 0x0000000000547ee5 in raise ()
#1 0x000000000050e635 in abort ()
#2 0x000000000051f365 in __libc_message ()
#3 0x0000000000524d22 in malloc_printerr ()
#4 0x0000000000526f4d in _int_malloc ()
#5 0x0000000000527b7b in malloc ()
#6 0x00000000004f0fcd in operator new(unsigned long) ()
#7 0x00000000004d2829 in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long,
std::allocator<char> const&) ()
#8 0x00000000004d329a in char* std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::_S_construct<char const*>(char const*, char const*,
std::allocator<char> const&, std::forward_iterator_tag) ()
#9 0x00000000004d340b in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::basic_string(char const*, unsigned long,
std::allocator<char> const&) ()
#10 0x0000000000495c39 in HMMTables<int,
WordClasses>::readJumps(std::basic_istream<char, std::char_traits<char> >&) ()
#11 0x000000000048bdf2 in hmm::load_table(char const*) ()
#12 0x000000000042822f in StartTraining(int&) ()
#13 0x00000000004294ba in main ()
Is it a problem with inc-Giza or could it be a problem with the libs being used.
Regards,
Sujay
_______________________________________________
Moses-support mailing list
[email protected]<mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
