I'm sorry I haven't had an opportunity to look into this yet (hopefully later this evening). But, one thing that you need to do is make sure that your config file has an entry setting the maximum phrase size to a very large number, which prevents some bad pruning from taking place that can lead to crashes.
[max-phrase-length] 9999 Chris On Mon, Aug 9, 2010 at 12:28 PM, Sylvain Raybaud <[email protected]> wrote: > Some more information before I leave on a two weeks break. > > I've run it on a different "real world" graph (test-07), it also crashes, > apparently at the same location but with "free(): invalid next size (normal): > 0x000000001db930d0" (see back trace attached). > > moses output (with -v 2) shows so called "shortest paths" which I would expect > to be computed only if the lattice is completely loaded > > if someone feels like running some tests he'll find attached: > test-07.slf.gz generated by sphinx > test-07.moses-verbose2.stderr.gz moses output > backtrace.gz > latest version of script > > when I come back I'll try to add more debug output to moses so that I can > pinpoint where the error occurs (I think this is gonna be painful since I've > never had a look in moses source code so far) > > see you > > Sylvain > > > On Monday 09 August 2010 15:56:31 Sylvain Raybaud wrote: >> Hello >> >> Here is the latest version of the script, which replace noise information >> with epsilon transitions. Moses successfully translates very small "toy" >> lattice generated with this script but on a real world example >> (http://perso.crans.org/raybaud/realworld.slf.gz) Moses fails with glibc >> complaining about a corrupted double-linked list: >> sylv...@markov /global/markov/raybauds/XP/S2T $ nice ~/softs/moses/moses- >> cmd/src/moses -max-phrase-length 100000 -inputtype 2 -weight-i 1 -f >> working- dir/filtered-OlliRehn-1/moses.ini < realworld.plf > >> realworld.plf.translated Defined parameters (per moses.ini or switch): >> config: working-dir/filtered-OlliRehn-1/moses.ini >> distortion-file: 0-0 msd-bidirectional-fe 6 >> /global/markov/raybauds/XP/S2T/working-dir/filtered-OlliRehn-1/reordering-t >> able distortion-limit: 6 >> input-factors: 0 >> inputtype: 2 >> lmodel-file: 0 0 5 /home/sylvain/GMR/SYSTEMS/WMT09/lm/europarl.lm >> mapping: 0 T 0 >> max-phrase-length: 100000 >> ttable-file: 0 0 5 >> /global/markov/raybauds/XP/S2T/working-dir/filtered- >> OlliRehn-1/phrase-table >> ttable-limit: 20 >> weight-d: 0.054012 0.196064 -0.003181 0.081071 0.086735 0.062332 >> 0.090617 >> weight-i: 1 >> weight-l: 0.064254 >> weight-t: 0.013876 0.081935 0.059305 0.013773 0.145934 >> weight-w: 0.046909 >> Loading lexical distortion models...have 1 models >> [...] >> Finished loading phrase tables : [13.000] seconds >> IO from STDOUT/STDIN >> Created input-output object : [13.000] seconds >> WARN: probability > 1: 1.492 >> *** glibc detected *** /home/sylvain/softs/moses/moses-cmd/src/moses: >> corrupted double-linked list: 0x000000001e7d8610 *** >> >> I used the very latest version, fresh trunk from this morning. Of course I >> can't say if this is because my lattices are ill formed or because of >> moses. So I recompiled it without optimisation (CFLAGS = -g -ggdb -O0, >> CXXFLAGS also) to get a full backtrace: >> >> #0 0x00007ffff70f81b5 in raise () from /lib/libc.so.6 >> #1 0x00007ffff70f95e0 in abort () from /lib/libc.so.6 >> #2 0x00007ffff71333d7 in __libc_message () from /lib/libc.so.6 >> #3 0x00007ffff7138966 in malloc_printerr () from /lib/libc.so.6 >> #4 0x00007ffff713a398 in _int_free () from /lib/libc.so.6 >> #5 0x00007ffff713d71c in free () from /lib/libc.so.6 >> #6 0x000000000040d820 in __gnu_cxx::new_allocator<unsigned >> long>::deallocate (this=0x1d8d7738, __p=0x1da04c60) at >> /usr/lib/gcc/x86_64-pc-linux- >> gnu/4.4.4/include/g++-v4/ext/new_allocator.h:95 >> #7 0x000000000040c45e in std::_Bvector_base<std::allocator<bool> >> >> >::_M_deallocate (this=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- >> >> gnu/4.4.4/include/g++-v4/bits/stl_bvector.h:444 >> #8 0x000000000040b6a1 in ~_Bvector_base (this=0x1d8d7738, __in_chrg=<value >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> v4/bits/stl_bvector.h:430 >> #9 0x000000000040a6d6 in ~vector (this=0x1d8d7738, __in_chrg=<value >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> v4/bits/stl_bvector.h:547 >> #10 0x00000000004b87df in std::_Destroy<std::vector<bool, >> std::allocator<bool> >> >> > > (__pointer=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- >> >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:83 >> #11 0x00000000004b82e1 in >> std::_Destroy_aux<false>::__destroy<std::vector<bool, std::allocator<bool> >> >*> (__first=0x1d8d7738, __last=0x1d8e56a8) >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> v4/bits/stl_construct.h:93 >> #12 0x00000000004b77ee in std::_Destroy<std::vector<bool, >> std::allocator<bool> >> >> >*> (__first=0x1d8c0470, __last=0x1d8e56a8) at >> >/usr/lib/gcc/x86_64-pc-linux- >> >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:116 >> #13 0x00000000004b68f9 in std::_Destroy<std::vector<bool, >> std::allocator<bool> >> >> >*, std::vector<bool, std::allocator<bool> > > (__first=0x1d8c0470, >> >> __last=0x1d8e56a8) >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> v4/bits/stl_construct.h:142 >> #14 0x00000000004b6030 in ~vector (this=0x7fffffffcc30, __in_chrg=<value >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> v4/bits/stl_vector.h:313 >> #15 0x00000000004b5141 in Moses::WordLattice::Read (this=0x1caf2d70, >> in=..., factorOrder=...) at WordLattice.cpp:110 >> #16 0x0000000000416d2b in IOWrapper::GetInput (this=0x1caf0760, >> inputType=0x1caf2d70) at IOWrapper.cpp:171 >> #17 0x0000000000418c1c in ReadInput (ioWrapper=..., >> inputType=Moses::WordLatticeInput, sour...@0x7fffffffcf48) at >> IOWrapper.cpp:511 #18 0x00000000004086c4 in main (argc=11, >> argv=0x7fffffffd0f8) at Main.cpp:398 >> >> As I understand it, it crashed while it reads the lattice. So there must be >> something wrong in my script. It can't be probability above one, can it? >> >> Any hint is more than welcome... >> >> regards, >> >> Sylvain >> >> On Friday 06 August 2010 11:44:25 Sylvain Raybaud wrote: >> > Hello >> > >> > A few days ago I came to you asking if someone knew of a script to turn >> > >> > word lattice from SLF (HTK format) into PLF (Moses input format). It >> > seems that doesn't exist yet, so here is my proposal (python file >> > attached). >> > >> > I tried to include all necessary information in help message and in file >> > header. >> > >> > basic use : >> > slf2plf input > output >> > slf2plf < input > output >> > slf2plf -h for help >> > >> > it works on small "toy" lattices, I checked the results. I've run it on >> > real world lattices but I have not checked the results yet. >> > >> > I tried to keep as much flexibility of SLF format as possible, but >> > sublattices won't work yet (they will be parsed but in the resulting >> > graph they will appear as a single node. I haven't checked this though). >> > >> > Also, for the moment the software requires that words are specified on >> > links (not nodes) and that acoustic and language model scores are >> > present. The resulting score in output graph is: >> > acoustic+language*lmscale >> > >> > all testing, remarks and requests welcome (of course). I can make the >> > script accessible on a mercurial repository if you want. >> > >> > Hope this helps >> > >> > regards, > > -- > Sylvain > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
