Yes, that will do it. On Mon, Aug 9, 2010 at 4:13 PM, Sylvain Raybaud <[email protected]> wrote: > On Monday 09 August 2010 20:43:53 Chris Dyer wrote: >> I'm sorry I haven't had an opportunity to look into this yet >> (hopefully later this evening). But, one thing that you need to do is >> make sure that your config file has an entry setting the maximum >> phrase size to a very large number, which prevents some bad pruning >> from taking place that can lead to crashes. >> >> [max-phrase-length] >> 9999 >> >> Chris >> > > Hi, thanks for your answer! I run moses with the option: > -max-phrase-length 100000 > > this should do, right? > > regards, > > Sylvain > >> On Mon, Aug 9, 2010 at 12:28 PM, Sylvain Raybaud >> >> <[email protected]> wrote: >> > Some more information before I leave on a two weeks break. >> > >> > I've run it on a different "real world" graph (test-07), it also crashes, >> > apparently at the same location but with "free(): invalid next size >> > (normal): 0x000000001db930d0" (see back trace attached). >> > >> > moses output (with -v 2) shows so called "shortest paths" which I would >> > expect to be computed only if the lattice is completely loaded >> > >> > if someone feels like running some tests he'll find attached: >> > test-07.slf.gz generated by sphinx >> > test-07.moses-verbose2.stderr.gz moses output >> > backtrace.gz >> > latest version of script >> > >> > when I come back I'll try to add more debug output to moses so that I can >> > pinpoint where the error occurs (I think this is gonna be painful since >> > I've never had a look in moses source code so far) >> > >> > see you >> > >> > Sylvain >> > >> > On Monday 09 August 2010 15:56:31 Sylvain Raybaud wrote: >> >> Hello >> >> >> >> Here is the latest version of the script, which replace noise >> >> information with epsilon transitions. Moses successfully translates >> >> very small "toy" lattice generated with this script but on a real world >> >> example >> >> (http://perso.crans.org/raybaud/realworld.slf.gz) Moses fails with glibc >> >> complaining about a corrupted double-linked list: >> >> sylv...@markov /global/markov/raybauds/XP/S2T $ nice >> >> ~/softs/moses/moses- cmd/src/moses -max-phrase-length 100000 -inputtype >> >> 2 -weight-i 1 -f working- dir/filtered-OlliRehn-1/moses.ini < >> >> realworld.plf > >> >> realworld.plf.translated Defined parameters (per moses.ini or switch): >> >> config: working-dir/filtered-OlliRehn-1/moses.ini >> >> distortion-file: 0-0 msd-bidirectional-fe 6 >> >> /global/markov/raybauds/XP/S2T/working-dir/filtered-OlliRehn-1/reorderin >> >> g-t able distortion-limit: 6 >> >> input-factors: 0 >> >> inputtype: 2 >> >> lmodel-file: 0 0 5 >> >> /home/sylvain/GMR/SYSTEMS/WMT09/lm/europarl.lm mapping: 0 T 0 >> >> max-phrase-length: 100000 >> >> ttable-file: 0 0 5 >> >> /global/markov/raybauds/XP/S2T/working-dir/filtered- >> >> OlliRehn-1/phrase-table >> >> ttable-limit: 20 >> >> weight-d: 0.054012 0.196064 -0.003181 0.081071 0.086735 0.062332 >> >> 0.090617 >> >> weight-i: 1 >> >> weight-l: 0.064254 >> >> weight-t: 0.013876 0.081935 0.059305 0.013773 0.145934 >> >> weight-w: 0.046909 >> >> Loading lexical distortion models...have 1 models >> >> [...] >> >> Finished loading phrase tables : [13.000] seconds >> >> IO from STDOUT/STDIN >> >> Created input-output object : [13.000] seconds >> >> WARN: probability > 1: 1.492 >> >> *** glibc detected *** /home/sylvain/softs/moses/moses-cmd/src/moses: >> >> corrupted double-linked list: 0x000000001e7d8610 *** >> >> >> >> I used the very latest version, fresh trunk from this morning. Of course >> >> I can't say if this is because my lattices are ill formed or because of >> >> moses. So I recompiled it without optimisation (CFLAGS = -g -ggdb -O0, >> >> CXXFLAGS also) to get a full backtrace: >> >> >> >> #0 0x00007ffff70f81b5 in raise () from /lib/libc.so.6 >> >> #1 0x00007ffff70f95e0 in abort () from /lib/libc.so.6 >> >> #2 0x00007ffff71333d7 in __libc_message () from /lib/libc.so.6 >> >> #3 0x00007ffff7138966 in malloc_printerr () from /lib/libc.so.6 >> >> #4 0x00007ffff713a398 in _int_free () from /lib/libc.so.6 >> >> #5 0x00007ffff713d71c in free () from /lib/libc.so.6 >> >> #6 0x000000000040d820 in __gnu_cxx::new_allocator<unsigned >> >> long>::deallocate (this=0x1d8d7738, __p=0x1da04c60) at >> >> /usr/lib/gcc/x86_64-pc-linux- >> >> gnu/4.4.4/include/g++-v4/ext/new_allocator.h:95 >> >> #7 0x000000000040c45e in std::_Bvector_base<std::allocator<bool> >> >> >> >> >::_M_deallocate (this=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- >> >> >> >> gnu/4.4.4/include/g++-v4/bits/stl_bvector.h:444 >> >> #8 0x000000000040b6a1 in ~_Bvector_base (this=0x1d8d7738, >> >> __in_chrg=<value optimized out>) at >> >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> >> v4/bits/stl_bvector.h:430 >> >> #9 0x000000000040a6d6 in ~vector (this=0x1d8d7738, __in_chrg=<value >> >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> >> v4/bits/stl_bvector.h:547 >> >> #10 0x00000000004b87df in std::_Destroy<std::vector<bool, >> >> std::allocator<bool> >> >> >> >> > > (__pointer=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- >> >> >> >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:83 >> >> #11 0x00000000004b82e1 in >> >> std::_Destroy_aux<false>::__destroy<std::vector<bool, >> >> std::allocator<bool> >> >> >> >> >*> (__first=0x1d8d7738, __last=0x1d8e56a8) >> >> >> >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> >> v4/bits/stl_construct.h:93 >> >> #12 0x00000000004b77ee in std::_Destroy<std::vector<bool, >> >> std::allocator<bool> >> >> >> >> >*> (__first=0x1d8c0470, __last=0x1d8e56a8) at >> >> >/usr/lib/gcc/x86_64-pc-linux- >> >> >> >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:116 >> >> #13 0x00000000004b68f9 in std::_Destroy<std::vector<bool, >> >> std::allocator<bool> >> >> >> >> >*, std::vector<bool, std::allocator<bool> > > (__first=0x1d8c0470, >> >> >> >> __last=0x1d8e56a8) >> >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> >> v4/bits/stl_construct.h:142 >> >> #14 0x00000000004b6030 in ~vector (this=0x7fffffffcc30, __in_chrg=<value >> >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- >> >> v4/bits/stl_vector.h:313 >> >> #15 0x00000000004b5141 in Moses::WordLattice::Read (this=0x1caf2d70, >> >> in=..., factorOrder=...) at WordLattice.cpp:110 >> >> #16 0x0000000000416d2b in IOWrapper::GetInput (this=0x1caf0760, >> >> inputType=0x1caf2d70) at IOWrapper.cpp:171 >> >> #17 0x0000000000418c1c in ReadInput (ioWrapper=..., >> >> inputType=Moses::WordLatticeInput, sour...@0x7fffffffcf48) at >> >> IOWrapper.cpp:511 #18 0x00000000004086c4 in main (argc=11, >> >> argv=0x7fffffffd0f8) at Main.cpp:398 >> >> >> >> As I understand it, it crashed while it reads the lattice. So there must >> >> be something wrong in my script. It can't be probability above one, can >> >> it? >> >> >> >> Any hint is more than welcome... >> >> >> >> regards, >> >> >> >> Sylvain >> >> >> >> On Friday 06 August 2010 11:44:25 Sylvain Raybaud wrote: >> >> > Hello >> >> > >> >> > A few days ago I came to you asking if someone knew of a script to >> >> > turn >> >> > >> >> > word lattice from SLF (HTK format) into PLF (Moses input format). It >> >> > seems that doesn't exist yet, so here is my proposal (python file >> >> > attached). >> >> > >> >> > I tried to include all necessary information in help message and in >> >> > file header. >> >> > >> >> > basic use : >> >> > slf2plf input > output >> >> > slf2plf < input > output >> >> > slf2plf -h for help >> >> > >> >> > it works on small "toy" lattices, I checked the results. I've run it >> >> > on real world lattices but I have not checked the results yet. >> >> > >> >> > I tried to keep as much flexibility of SLF format as possible, but >> >> > sublattices won't work yet (they will be parsed but in the resulting >> >> > graph they will appear as a single node. I haven't checked this >> >> > though). >> >> > >> >> > Also, for the moment the software requires that words are specified on >> >> > links (not nodes) and that acoustic and language model scores are >> >> > present. The resulting score in output graph is: >> >> > acoustic+language*lmscale >> >> > >> >> > all testing, remarks and requests welcome (of course). I can make the >> >> > script accessible on a mercurial repository if you want. >> >> > >> >> > Hope this helps >> >> > >> >> > regards, >> > >> > -- >> > Sylvain >> > >> > _______________________________________________ >> > Moses-support mailing list >> > [email protected] >> > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- > Sylvain > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
