Re: [Moses-support] phrase-table.gz - 0(Zero) Bytes
Hi, there is this error message: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate This binary should have been compiled with all the other code. Please check you compilation for errors. -phi On Thu, Nov 15, 2012 at 11:06 PM, Don Erick Bonus wrote: > Hi. We have tried the baseline example from the Moses website. However, > during the Training the Translation System stage, and as we trace the > process/logs of training.out we get the final set of lines. > > Finished Thu Nov 15 20:00:08 2012 > (6.6) consolidating the two halves @ Thu Nov 15 20:00:08 PHT 2012 > Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate > ./model/phrase-table.half.f2e.gz ./model/phrase-table.half.e2f.gz > /dev/stdout | gzip -c > ./model/phrase-table.gz > sh: 1: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate: not found > Executing: rm -f ./model/phrase-table.half.* > (7) learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 > (7.1) [no factors] learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 > (7.2) building tables @ Thu Nov 15 20:00:08 PHT 2012 > Executing: > /home/ccsadmin/mosesdecoder/scripts/../bin/lexical-reordering-score > ./model/extract.o.sorted.gz 0.5 ./model/reordering-table. --model "wbe msd > wbe-msd-bidirectional-fe" > Lexical Reordering Scorer > scores lexical reordering models of several types (hierarchical, > phrase-based and word-based-extraction > (8) learn generation model @ Thu Nov 15 20:00:08 PHT 2012 > no generation model requested, skipping step > (9) create moses.ini @ Thu Nov 15 20:00:08 PHT 2012 > > The case is almost the same with our own training data. Upon cheking on the > produced phrase-table.gz we get 0 bytes. > > Please advice... > > Thanks. > > Erick > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] How to set up Moses Online Demonstrations
Hi there, you can run Moses in server mode which would make available its translation capabilities via XML-RPC; The Moses manual is available here: - http://www.statmt.org/moses/manual/manual.pdf See the section on "Moses Server" on page 116 for more information. There's also a Wiki page from Herve Saint-Amand on web translation: - http://www.statmt.org/moses/?n=Moses.WebTranslation In 2010, I released MT Server Land, which basically allows to run Moses in a "web service" fashion -- source code is available here: - https://github.com/cfedermann/mt-serverland Hope that helps, best regards, Christian -- Dipl.-Inf. Christian Federmann, Researcher, Language Technology Lab Office +1.09 -- Phone +49-681/857-75-5353, Fax +49-681/857-75-5338 DFKI GmbH, Campus D3 2, Stuhlsatzenhausweg 3, 66123 Saarbruecken http://www.dfki.de/~cfedermann --- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 --- On 16.11.2012, at 03:58, dmp wrote: >Hello, I am a student and I want to set up Moses Online Demostrtions. But > I don't know how to build it. I search it in your sit but I just find a > little message about it. In " Web Translation " and " Web Interface " ,there > is some information about that. But I also have problem. > 1. How to use moses to set up a system just like "google translation". > 2.Does moses Support web function? Or I should write script to Handle the > requst from web. > 3.If moses support web function,how can I use it. >would you give me detailed answers. Or give some document about it. Thank > you very much. > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] phrase-table.gz - 0(Zero) Bytes
Hi. We have tried the baseline example from the Moses website. However, during the Training the Translation System stage, and as we trace the process/logs of training.out we get the final set of lines. Finished Thu Nov 15 20:00:08 2012 (6.6) consolidating the two halves @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate ./model/phrase-table.half.f2e.gz ./model/phrase-table.half.e2f.gz /dev/stdout | gzip -c > ./model/phrase-table.gz sh: 1: /home/ccsadmin/mosesdecoder/scripts/../bin/consolidate: not found Executing: rm -f ./model/phrase-table.half.* (7) learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.1) [no factors] learn reordering model @ Thu Nov 15 20:00:08 PHT 2012 (7.2) building tables @ Thu Nov 15 20:00:08 PHT 2012 Executing: /home/ccsadmin/mosesdecoder/scripts/../bin/lexical-reordering-score ./model/extract.o.sorted.gz 0.5 ./model/reordering-table. --model "wbe msd wbe-msd-bidirectional-fe" Lexical Reordering Scorer scores lexical reordering models of several types (hierarchical, phrase-based and word-based-extraction (8) learn generation model @ Thu Nov 15 20:00:08 PHT 2012 no generation model requested, skipping step (9) create moses.ini @ Thu Nov 15 20:00:08 PHT 2012 The case is almost the same with our own training data. Upon cheking on the produced phrase-table.gz we get 0 bytes. Please advice... Thanks. Erick___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] How to set up Moses Online Demonstrations
Hello, I am a student and I want to set up Moses Online Demostrtions. But I don't know how to build it. I search it in your sit but I just find a little message about it. In " Web Translation " and " Web Interface " ,there is some information about that. But I also have problem. 1. How to use moses to set up a system just like "google translation". 2.Does moses Support web function? Or I should write script to Handle the requst from web. 3.If moses support web function,how can I use it. would you give me detailed answers. Or give some document about it. Thank you very much. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
OK. Fixed. Hieu messed up the alignment :P But in his defense, I was looking directly at it and did not see it for some time :) W dniu 16.11.2012 16:39, Marcin Junczys-Dowmunt pisze: > OK, confirmed. The alignment points do not vanish with "-encoding None". > > But this still seems to be a bug, the more efficient compression should > not lose that many alignment points, this should only happen very rarely > if at all. > > W dniu 16.11.2012 16:27, Marcin Junczys-Dowmunt pisze: >> Hi, >> this might actually be a consequence of the compression method I use. >> You can also check and recompact the phrase table with the following option >> >> -encoding None >> >> It will be larger by some 20-30%, but it should not be missing alignment >> points. If it still does then this is an implementation mistake I have >> to fix. >> >> I will check your data now. >> >> W dniu 16.11.2012 16:22, Rico Sennrich pisze: >>> On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: can you put the sample data somewhere public like dropbox or googgle drive. i'll take a look at it as well >>> the table is on >>> https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ >>> >>> the command I used to build the compact table: >>> >>> ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out >>> sample_data -alignment-info -rankscore 0 -nscores 3 >>> >>> >>> To reproduce the problem, add a check to assert that alignment.size() > >>> 0 in PhraseTableCreator::EncodeAlignment() . >>> >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
OK, confirmed. The alignment points do not vanish with "-encoding None". But this still seems to be a bug, the more efficient compression should not lose that many alignment points, this should only happen very rarely if at all. W dniu 16.11.2012 16:27, Marcin Junczys-Dowmunt pisze: > Hi, > this might actually be a consequence of the compression method I use. > You can also check and recompact the phrase table with the following option > > -encoding None > > It will be larger by some 20-30%, but it should not be missing alignment > points. If it still does then this is an implementation mistake I have > to fix. > > I will check your data now. > > W dniu 16.11.2012 16:22, Rico Sennrich pisze: >> On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: >>> can you put the sample data somewhere public like dropbox or googgle >>> drive. >>> >>> i'll take a look at it as well >>> >> the table is on >> https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ >> >> the command I used to build the compact table: >> >> ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out >> sample_data -alignment-info -rankscore 0 -nscores 3 >> >> >> To reproduce the problem, add a check to assert that alignment.size() > >> 0 in PhraseTableCreator::EncodeAlignment() . >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
Hi, this might actually be a consequence of the compression method I use. You can also check and recompact the phrase table with the following option -encoding None It will be larger by some 20-30%, but it should not be missing alignment points. If it still does then this is an implementation mistake I have to fix. I will check your data now. W dniu 16.11.2012 16:22, Rico Sennrich pisze: > On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: >> can you put the sample data somewhere public like dropbox or googgle >> drive. >> >> i'll take a look at it as well >> > the table is on > https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ > > the command I used to build the compact table: > > ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out > sample_data -alignment-info -rankscore 0 -nscores 3 > > > To reproduce the problem, add a check to assert that alignment.size() > > 0 in PhraseTableCreator::EncodeAlignment() . > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact phrase table: missing alignments
On Fri, 2012-11-16 at 15:08 +, Hieu Hoang wrote: > can you put the sample data somewhere public like dropbox or googgle > drive. > > i'll take a look at it as well > the table is on https://docs.google.com/open?id=0B5nFI-wykGgneDFVNFFfNVZFYzQ the command I used to build the compact table: ~/mosesdecoder/bin/processPhraseTableMin -in sample_data.gz -out sample_data -alignment-info -rankscore 0 -nscores 3 To reproduce the problem, add a check to assert that alignment.size() > 0 in PhraseTableCreator::EncodeAlignment() . ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
Just to let you know, I am currently looking at the phrase table. Several things seem to be off in the latest version, some of them seem to be related to sparse moses, some not. I will report back once I think I got through them. W dniu 16.11.2012 12:24, Barry Haddow pisze: > Hi Rico > > You need to remove the ".minlexr" from the end of the reordering table > path. Moses is checking for a path ending with .minlexr.minlexr, which > doesn't exist, so it's trying to load a memory-based table. > > I was hitting the same problem, although now I get the error: > > Using compact phrase table > tcmalloc: large alloc 13312782167009714176 bytes == (nil) @ > > when trying to load the compact reordering table, > > cheers - Barry > > On 16/11/12 09:36, Rico Sennrich wrote: >> Hi Marcin, hi list, >> >> I can't load compact lexical reordering models. I don't know if it's >> generally broken or if you need some more information, but here's all >> the info I have without digging around in the code. I created it with >> the following settings, and there were no errors: >> >> mosesdecoder/bin/processLexicalTableMin -in >> reordering-table.wbe-msd-bidirectional-fe.gz -out >> reordering-table.wbe-msd-bidirectional-fe -threads 12 >> >> The config looks like this: >> >> # distortion (reordering) files >> [distortion-file] >> 0-0 wbe-msd-bidirectional-fe-allff >> 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr >> >> and this is the relevant bit of stderr: >> >> ScoreProducer: Distortion start: 0 end: 1 >> ScoreProducer: WordPenalty start: 1 end: 2 >> ScoreProducer: !UnknownWordPenalty start: 2 end: 3 >> Loading lexical distortion models...have 1 models >> ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 >> end: 9 >> Creating lexical reordering... >> weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 >> Loading table into memory...terminate called after throwing an instance >> of 'std::out_of_range' >> what(): vector::_M_range_check >> Aborted >> >> Using yesterday's git commit (35cb7e6). >> >> best, >> Rico >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] compact phrase table: missing alignments
Hi again, my last problems with the compact tables was a configuration problem, but I think this one isn't. I'm working on stuff that relies on word alignment during decoding, but for some phrase pairs, targetPhrase->GetAlignTerm() is missing alignment pairs in compact models (everything is dandy with in-memory tables). I traced the problem back to the variable "a" in PhraseTableCreator::EncodeLine(), which stores the alignment. Somewhere between its being filled with (lines 736-742) and the call to EncodeAlignment(), alignments go missing. Specifically, this happens in EncodeTargetPhrasePREnc (lines 639-645). I have no idea what the purpose of this bit is, so I won't touch it. But it would be great to have the full alignment information during decoding. I can send you a sample phrase table (4kb) to reproduce the problem. best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
Hi Rico You need to remove the ".minlexr" from the end of the reordering table path. Moses is checking for a path ending with .minlexr.minlexr, which doesn't exist, so it's trying to load a memory-based table. I was hitting the same problem, although now I get the error: Using compact phrase table tcmalloc: large alloc 13312782167009714176 bytes == (nil) @ when trying to load the compact reordering table, cheers - Barry On 16/11/12 09:36, Rico Sennrich wrote: > Hi Marcin, hi list, > > I can't load compact lexical reordering models. I don't know if it's > generally broken or if you need some more information, but here's all > the info I have without digging around in the code. I created it with > the following settings, and there were no errors: > > mosesdecoder/bin/processLexicalTableMin -in > reordering-table.wbe-msd-bidirectional-fe.gz -out > reordering-table.wbe-msd-bidirectional-fe -threads 12 > > The config looks like this: > > # distortion (reordering) files > [distortion-file] > 0-0 wbe-msd-bidirectional-fe-allff > 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr > > and this is the relevant bit of stderr: > > ScoreProducer: Distortion start: 0 end: 1 > ScoreProducer: WordPenalty start: 1 end: 2 > ScoreProducer: !UnknownWordPenalty start: 2 end: 3 > Loading lexical distortion models...have 1 models > ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 > end: 9 > Creating lexical reordering... > weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 > Loading table into memory...terminate called after throwing an instance > of 'std::out_of_range' >what(): vector::_M_range_check > Aborted > > Using yesterday's git commit (35cb7e6). > > best, > Rico > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
On Fri, 2012-11-16 at 12:02 +0100, Marcin Junczys-Dowmunt wrote: > Hi Rico, > I will take a look today. Just, to be sure: are you using a 64-bit machine? yes: uname -r 2.6.35.14-106.fc14.x86_64 ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] compact lexical reordering model crashes
Hi Rico, I will take a look today. Just, to be sure: are you using a 64-bit machine? W dniu 16.11.2012 10:36, Rico Sennrich pisze: > Hi Marcin, hi list, > > I can't load compact lexical reordering models. I don't know if it's > generally broken or if you need some more information, but here's all > the info I have without digging around in the code. I created it with > the following settings, and there were no errors: > > mosesdecoder/bin/processLexicalTableMin -in > reordering-table.wbe-msd-bidirectional-fe.gz -out > reordering-table.wbe-msd-bidirectional-fe -threads 12 > > The config looks like this: > > # distortion (reordering) files > [distortion-file] > 0-0 wbe-msd-bidirectional-fe-allff > 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr > > and this is the relevant bit of stderr: > > ScoreProducer: Distortion start: 0 end: 1 > ScoreProducer: WordPenalty start: 1 end: 2 > ScoreProducer: !UnknownWordPenalty start: 2 end: 3 > Loading lexical distortion models...have 1 models > ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 > end: 9 > Creating lexical reordering... > weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 > Loading table into memory...terminate called after throwing an instance > of 'std::out_of_range' >what(): vector::_M_range_check > Aborted > > Using yesterday's git commit (35cb7e6). > > best, > Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] compact lexical reordering model crashes
Hi Marcin, hi list, I can't load compact lexical reordering models. I don't know if it's generally broken or if you need some more information, but here's all the info I have without digging around in the code. I created it with the following settings, and there were no errors: mosesdecoder/bin/processLexicalTableMin -in reordering-table.wbe-msd-bidirectional-fe.gz -out reordering-table.wbe-msd-bidirectional-fe -threads 12 The config looks like this: # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /path/to/model/reordering-table.wbe-msd-bidirectional-fe.minlexr and this is the relevant bit of stderr: ScoreProducer: Distortion start: 0 end: 1 ScoreProducer: WordPenalty start: 1 end: 2 ScoreProducer: !UnknownWordPenalty start: 2 end: 3 Loading lexical distortion models...have 1 models ScoreProducer: LexicalReordering_wbe-msd-bidirectional-fe-allff start: 3 end: 9 Creating lexical reordering... weights: 0.048479 0.0534099 0.111232 0.0528042 0.00404942 0.0169295 Loading table into memory...terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check Aborted Using yesterday's git commit (35cb7e6). best, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
