if you're interested in the fuzzy match rule table, you may be interested to know it is now multithreaded.
>From what i see, it works well for 2 threads, but there is a rapidly diminishing rate of return after that. Some results for translating 100 sentences: 1 2 3 4 load 138.149 137.893 138.538 137.821 just decoding 47.09 28.62 24.258 24.095 end 185.239 166.513 162.796 161.916 If you use it, please give me some feedback on what you think and how it can be improved On 8 October 2012 15:41, <[email protected]> wrote: > Hello, > > when starting to test the "Fuzzy Match Rule Table for Hierachical Models" > as described, I get a segfault. Probably I'm doing something stupid or > there's simply something wrong in my ini file or my boost installation, but > maybe someone wants to have a short look and enlighten me. I'm running > version 24a6425b23 from end of September (27th, as far as I remember). > My ini file is attached, and a gdb session protocol below. > > Please let me know if you need any further information. > > BTW: It seems that moses creates small files called for instance > 'mosesj2VNXl' in this setup. Is that normal? Will I get one per sentence, > and would I have to remove them or will they disappear after translation? > > Thanks > Stephan > > > ####### > [waltest@achird package]$ gdb `which moses` > GNU gdb Red Hat Linux (6.5-16.el5rh) > Copyright (C) 2006 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host > libthread_db library "/lib64/libthread_db.so.1". > > (gdb) set args -f ./moses.ini > (gdb) run > Starting program: > /ec/dgt/shared/exodus/MosesSuite_installations/MosesSuite_24a6425b23_2012-10-03/bin/moses > -f ./moses.ini > [Thread debugging using libthread_db enabled] > [New process 22163] > Defined parameters (per moses.ini or switch): > config: ./moses.ini > distortion-limit: 6 > input-factors: 0 > lmodel-file: 8 0 7 ./model/lm/lm > mapping: 0 T 0 > ttable-file: 11 0 0 3 > ../prepared-corpus/training-corpus.clean.300.preprocess.tok.lowercase.clean.en;../prepared-corpus/ > training-corpus.clean.300.preprocess.tok.lowercase.clean.fr > ;../alignment/aligned/aligned.grow-diag-final-and > ttable-limit: 20 > weight-d: 0.6 > weight-l: 0.09207 > weight-t: 0.09506 0.0248862 0.143999 0.0355093 0.0023145 > weight-w: -0.289771 > > /ec/dgt/shared/exodus/MosesSuite_installations/MosesSuite_24a6425b23_2012-10-03/bin > Loading lexical distortion models...have 0 models > Start loading LanguageModel ./model/lm/lm : [0.013] seconds > Finished loading LanguageModels : [0.088] seconds > Start loading PhraseTable > ../prepared-corpus/training-corpus.clean.300.preprocess.tok.lowercase.clean.en;../prepared-corpus/ > training-corpus.clean.300.preprocess.tok.lowercase.clean.fr;../alignment/aligned/aligned.grow-diag-final-and > : [0.088] seconds > filePath: > ../prepared-corpus/training-corpus.clean.300.preprocess.tok.lowercase.clean.en;../prepared-corpus/ > training-corpus.clean.300.preprocess.tok.lowercase.clean.fr > ;../alignment/aligned/aligned.grow-diag-final-and > Finished loading phrase tables : [0.088] seconds > IO from STDOUT/STDIN > Created input-output object : [0.088] seconds > [New Thread 1092372800 (LWP 22175)] > [New Thread 47398886633264 (LWP 22163)] > this is a small test . > Translating line 0 in thread id 1092372800 > Translating: this is a small test . > > > initStr=../prepared-corpus/training-corpus.clean.300.preprocess.tok.lowercase.clean.en;../prepared-corpus/ > training-corpus.clean.300.preprocess.tok.lowercase.clean.fr > ;../alignment/aligned/aligned.grow-diag-final-and > loading > ../prepared-corpus/training-corpus.clean.300.preprocess.tok.lowercase.clean.en > loading ../prepared-corpus/ > training-corpus.clean.300.preprocess.tok.lowercase.clean.fr > loading ../alignment/aligned/aligned.grow-diag-final-and > creating suffix array > 2583 words (incl. sentence boundaries) > done reading 2583 words, 300 sentences. > done sorting > done creating suffix array > loading mosestMqRR4 > 1 matches in 1 sentences. > reduced best cost from 2 to 2 > tm considered: 1 word-matched: 0 word-matched2: 0 best: 0 > pruned matches: nan > elapsed: 0 ( range: 0 match: 0 tm: 0 (validation: 0) ) > 9/12 (2/4) ||| -1 ||| > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 1092372800 (LWP 22175)] > 0x0000003a1309c8be in std::basic_string<char, std::char_traits<char>, > std::allocator<char> >::basic_string () from /usr/lib64/libstdc++.so.6 > (gdb) backtrace > #0 0x0000003a1309c8be in std::basic_string<char, std::char_traits<char>, > std::allocator<char> >::basic_string () from /usr/lib64/libstdc++.so.6 > #1 0x000000000051b8cb in tmmt::FuzzyMatchWrapper::create_extract > (this=0x1c6f2220, sentenceInd=0, cost=2, sourceSentence=@0x1c70e178, > targets=@0x1c733288, inputStr=@0x411c2e80, path=@0x411c2e60, > outputFile=@0x411c2ad0) > at > /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/basic_string.h:2106 > #2 0x000000000051ff15 in tmmt::FuzzyMatchWrapper::ExtractTM > (this=0x1c6f2220, inputPath=<value optimized out>) > at moses/src/fuzzy-match/FuzzyMatchWrapper.cpp:390 > #3 0x00000000005203bf in tmmt::FuzzyMatchWrapper::Extract > (this=0xfff950f0, inputPath=@0x1c721b80) at > moses/src/fuzzy-match/FuzzyMatchWrapper.cpp:52 > #4 0x0000000000505a59 in > Moses::PhraseDictionaryFuzzyMatch::InitializeForInput (this=0x1c6f2000, > inputSentence=@0x1c6f1110) > at moses/src/RuleTable/PhraseDictionaryFuzzyMatch.cpp:99 > #5 0x00000000004f5a23 in Moses::PhraseDictionaryFeature::InitDictionary > (this=0x1c6efde0, system=0xffffffff, source=@0x1c6f1110) > at moses/src/PhraseDictionary.cpp:260 > #6 0x000000000049da72 in > Moses::TranslationSystem::InitializeBeforeSentenceProcessing > (this=0x1c6f08d8, source=@0x1c6f1110) > at moses/src/TranslationSystem.cpp:121 > #7 0x0000000000442834 in Manager (this=0x411c3d50, lineNumber=0, > source=@0x1c6f1110, searchAlgorithm=<value optimized out>, system=<value > optimized out>) > at moses/src/Manager.cpp:62 > #8 0x00000000004108d0 in MosesCmd::TranslationTask::Run (this=0x1c6f0780) > at moses-cmd/src/Main.cpp:114 > #9 0x00000000004ae846 in Moses::ThreadPool::Execute (this=0x7fff8ed976b0) > at moses/src/ThreadPool.cpp:58 > #10 0x00002b1be96a0914 in thread_proxy () > from > /ec/dgt/shared/exodus/MosesSuite_installations/MosesSuite_24a6425b23_2012-10-03/deps/lib/libboost_thread-mt.so.1.46.1 > #11 0x00002b1be9ab873d in start_thread () from /lib64/libpthread.so.0 > #12 0x0000003a128d3d1d in clone () from /lib64/libc.so.6 > #13 0x0000000000000000 in ?? () > (gdb) quit > The program is running. Exit anyway? (y or n) y > [waltest@achird package]$ > > Stephan Walter > Directorate-General for Translation (DGT) > R.3 - Informatics > JMO A/2 003 > Rue Alcide de Gasperi > L-2920 Luxembourg > Tel.: (+352) 4301-31806 > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
