Hello Hieu,
I actually used the chart decoder and scripts to train the recaser and
consequently I used the chart decoder for the recaser decoding as well.
It seems that the chart decoder still writes an old-style moses.ini when
used with the recaser training scripts, however it's not able to read them.
In Moses::StaticData::LoadPhraseTables in StaticData.cpp:832
string filePath= token[4];
it expects [ttable-file] to have at least 5 entries, hardcoded. So the
recaser scripts are actually not usable with the chart decoder. You may
want to add an assertion for that in the code.
As a side note, I saw that the chart decoder always links to libpthread,
regardless of --enable-threads. Is the chart decoder multithreaded? and
can it still be used with the irstlm language model?
best regards
Christof
Hieu Hoang wrote:
> hi christof
>
> there's small fiddly changes to the ini file format so the chart decoder
> isn't backwardly compatible. You should use the trunk decoder if the
> data was trained using the trunk scripts
>
> On 22/01/2010 00:20, Christof Pintaske wrote:
>> Hi,
>>
>> is the moses/moses_chart executable from the mt3_chart branch supposed
>> to be usable as a recaser (that is using a phrase-table) ? When I do so
>> then I get a SEGV (see stacktrace at the very end).
>>
>> Working on the same files with moses from svn/trunk works fine.
>>
>> I read somewhere that irstlm is not thread-safe and in the debug output
>> I see that a new thread has been started. Is that the problem? I thought
>> I'd be safe because I did *not* configure moses with --enable-threads
>> nor with boost.
>>
>> many thanks
>> Christof
>>
>>
>>
>>
>> (gdb) run -f ~/data/engine/en-fr_chart/recaser/moses.ini< x
>> Starting program: /export/home/moses/libexec/moses-chart/bin/moses -f
>> ~/data/engine/en-fr_chart/recaser/moses.ini< x
>> [Thread debugging using libthread_db enabled]
>> Defined parameters (per moses.ini or switch):
>> config: /export/home/moses/data/engine/en-fr_chart/recaser/moses.ini
>> distortion-limit: 6
>> input-factors: 0
>> lmodel-file: 1 0 3
>> /export/home/moses/data/engine/en-fr_chart/recaser/cased.irstlm.gz
>> mapping: 0 T 0
>> ttable-file: 0 0 5
>> /export/home/moses/data/engine/en-fr_chart/recaser/phrase-table.gz
>> ttable-limit: 20
>> weight-d: 0.6
>> weight-l: 0.5000
>> weight-t: 0.2 0.2 0.2 0.2 0.2
>> weight-w: -1
>> Added 0 Distortion 0-0
>> Added 1 !UnknownWordPenalty 1-1
>> Added 2 WordPenalty 2-2
>> Loading lexical distortion models...
>> have 0 models
>> Start loading LanguageModel
>> /export/home/moses/data/engine/en-fr_chart/recaser/cased.irstlm.gz :
>> [0.000] seconds
>> Added 3 LanguageModel 3-3
>> In LanguageModelIRST::Load: nGramOrder = 3
>> Loading LM file (no MAP)
>> iARPA
>> loadtxt()
>> 1-grams: reading 8178 entries
>> 2-grams: reading 37042 entries
>> Detaching after fork from child process 4881.
>> 3-grams: reading 61742 entries
>> Detaching after fork from child process 4882.
>> done
>> OOV code is 8177
>> OOV code is 8177
>> IRST: m_unknownId=8177
>> Finished loading LanguageModels : [1.000] seconds
>> [New Thread 0x2b0788eaa940 (LWP 4878)]
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x000000398a29c8c8 in std::basic_string<char, std::char_traits<char>,
>> std::allocator<char> >::basic_string () from /usr/lib64/libstdc++.so.6
>> (gdb) where
>> #0 0x000000398a29c8c8 in std::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >::basic_string () from
>> /usr/lib64/libstdc++.so.6
>> #1 0x000000000045ca7a in Moses::StaticData::LoadPhraseTables
>> (this=0x74c8e0) at StaticData.cpp:832
>> #2 0x00000000004646ea in Moses::StaticData::LoadData (this=0x74c8e0,
>> parameter=<value optimized out>) at StaticData.cpp:406
>> #3 0x0000000000406fd1 in main (argc=3, argv=0x7fffeccfc5d8) at
>> ../../moses/src/StaticData.h:217
>> _______________________________________________
>> Moses-support mailing list
>> [email protected]
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
