Dear all,
I keep having the "decoder died" problem when running mert-moses(details
below). The decoder died in the middle of the process, e.g. at run 2, 8, 13,
... When I rerun the mert-mose with "--continue", normally it runs fine.
Maybe someone could help enlighten me on what could possible cause the
problem? I ofter run multiple mert-moses at the same time on around 4-6
corpora, so I'm not sure whether the problem below is due to the machine
resource being overloaded. (I attached the system "top" information here as
well).
*The error message, I normally encounter is:*
sh: line 1: 12100 Segmentation fault
/home/lmthang/HYP/src/moses/moses-cmd/src/moses -config filtered/moses.ini
-inputtype 0 -w -0.687588 -lm 0.009388 -d 0.039359 0.083356 0.000572
-0.007485 0.003778 -0.023522 -0.046637 -tm -0.005330 0.024834 0.064540
0.000418 0.003193 -n-best-list run2.best100.out 100 -i
/mnt/homes/lmthang/HYP/scripts/acl05_full_10/w-m_100/acl05_full.dev.processed.en
> run2.out
Exit code: 139
The decoder died. at
/home/lmthang/HYP/bin_02_June_08/moses-scripts/scripts-20080602-2352/training/mert-moses.pl
line 772.
*Sometimes, I saw the more detailed error here:*
*** glibc detected *** /home/lmthang/HYP/src/moses/moses-cmd/src/moses:
free(): invalid pointer: 0x3599f890 ***
======= Backtrace: =========
/lib/libc.so.6[0xab3b16]
/lib/libc.so.6(cfree+0x90)[0xab7070]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xf7f52151]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e977]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x806e965]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x8064c43]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x8061850]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x80621b8]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x809eb92]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses[0x8095b7a]
/lib/libc.so.6(exit+0xe9)[0xa769d9]
/lib/libc.so.6(__libc_start_main+0xe4)[0xa60df4]
/home/lmthang/HYP/src/moses/moses-cmd/src/moses(__gxx_personality_v0+0xbd)[0x804bc81]
======= Memory map: ========
00a2d000-00a47000 r-xp 00000000 09:02 15171585
/lib/ld-2.5.so
00a47000-00a48000 r--p 00019000 09:02 15171585
/lib/ld-2.5.so
00a48000-00a49000 rw-p 0001a000 09:02 15171585
/lib/ld-2.5.so
00a4b000-00b88000 r-xp 00000000 09:02 8716308
/lib/libc-2.5.so
00b88000-00b8a000 r--p 0013c000 09:02 8716308
/lib/libc-2.5.so
00b8a000-00b8b000 rw-p 0013e000 09:02 8716308
/lib/libc-2.5.so
00b8b000-00b8e000 rw-p 00b8b000 00:00 0
00ba9000-00bce000 r-xp 00000000 09:02 8716346
/lib/libm-2.5.so
00bce000-00bcf000 r--p 00024000 09:02 8716346
/lib/libm-2.5.so
00bcf000-00bd0000 rw-p 00025000 09:02 8716346
/lib/libm-2.5.so
00cd2000-00cdd000 r-xp 00000000 09:02 8717060
/lib/libgcc_s-4.1.2-20080102.so.1
00cdd000-00cde000 rw-p 0000a000 09:02 8717060
/lib/libgcc_s-4.1.2-20080102.so.1
00d12000-00d24000 r-xp 00000000 09:02 10961094
/usr/lib/libz.so.1.2.3
00d24000-00d25000 rw-p 00011000 09:02 10961094
/usr/lib/libz.so.1.2.3
08048000-08114000 r-xp 00000000 00:15 105283593
/mnt/homes/lmthang/HYP/src/moses/moses-cmd/src/moses
08114000-08115000 rw-p 000cb000 00:15 105283593
/mnt/homes/lmthang/HYP/src/moses/moses-cmd/src/moses
08115000-085ac000 rw-p 08115000 00:00 0
095db000-e57c7000 rw-p 095db000 00:00 0
f7d00000-f7d21000 rw-p f7d00000 00:00 0
f7d21000-f7e00000 ---p f7d21000 00:00 0
f7e9f000-f7ea1000 rw-p f7e9f000 00:00 0
f7ea1000-f7f7f000 r-xp 00000000 09:02 10965054
/usr/lib/libstdc++.so.6.0.8
f7f7f000-f7f82000 r--p 000dd000 09:02 10965054
/usr/lib/libstdc++.so.6.0.8
f7f82000-f7f84000 rw-p 000e0000 09:02 10965054
/usr/lib/libstdc++.so.6.0.8
f7f84000-f7f8a000 rw-p f7f84000 00:00 0
f7f9f000-f7fa1000 rw-p f7f9f000 00:00 0
ffe9d000-ffea1000 rw-p ffe9d000 00:00 0
[stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
sh: line 1: 11579 Aborted
/home/lmthang/HYP/src/moses/moses-cmd/src/moses -config filtered/moses.ini
-inputtype 0 -w -0.427595 -lm 0.010922 -d 0.099354 0.032944 -0.001238
-0.170074 0.041642 0.081615 -0.033839 -tm 0.003272 0.025515 0.030671
0.029248 -0.012071 -n-best-list run2.best100.out 100 -i
/mnt/homes/lmthang/HYP/scripts/acl05_full_10/w-m_10/acl05_full.dev.processed.en
> run2.out
Exit code: 134
The decoder died. at
/home/lmthang/HYP/bin_02_June_08/moses-scripts/scripts-20080602-2352/training/mert-moses.pl
line 772.
Thank you,
Regards,
Thang
--
Luong Minh Thang
WING group, School of Computing, National University of Singapore
http://wing.comp.nus.edu.sg/~lmthang
top - 11:15:47 up 46 days, 21:55, 2 users, load average: 11.31, 11.48, 10.61
Tasks: 197 total, 6 running, 191 sleeping, 0 stopped, 0 zombie
Cpu(s): 85.9%us, 0.5%sy, 0.0%ni, 1.8%id, 11.7%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8178368k total, 8130592k used, 47776k free, 1504k buffers
Swap: 66412688k total, 10292444k used, 56120244k free, 196780k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19774 lmthang 25 0 83780 80m 820 R 100 1.0 2:02.75 GIZA++
19686 lmthang 25 0 90792 87m 820 R 98 1.1 6:00.58 GIZA++
19483 lmthang 19 0 162m 12m 800 R 92 0.2 24:17.36 score-nbest.py
19521 lmthang 18 0 2011m 524m 816 R 67 6.6 10:16.51 moses
18924 lmthang 18 0 589m 307m 816 R 64 3.9 72:39.32 moses
19642 lmthang 18 0 3766m 2.5g 1080 D 1 31.8 6:50.91 moses
19482 lmthang 18 0 61624 2516 312 S 0 0.0 0:51.54 sort
19562 lmthang 18 0 3510m 1.0g 732 D 0 13.2 6:04.31 moses
19762 lmthang 15 0 30780 2192 1520 R 0 0.0 0:00.69 top
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
