Hi, if you have more data, you should train an entirely new language model, translation model, and tune again to get the most out of it.
-phi On Wed, Sep 24, 2008 at 10:44 AM, Panos <[EMAIL PROTECTED]> wrote: > Thanks for your reply. > Yes, that's what I mean by "enrich". > So, all I have to do is to start a new training process, with the new > corpus, but with the same language model? And what about the phrase table? > Will it just be updated with more entries? > > Thanks. > > Panagiotis > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Miles > Osborne > Sent: Wednesday, September 24, 2008 11:51 AM > To: Panos > Cc: Hieu Hoang; [email protected] > Subject: Re: [Moses-support] keep lm and phrase table in memory/incremental > training possible? > > if by "enrich" you mean add more entries, then yes, it is certainly > possible to do this, both for the LM and also the PT. > > depending upon the extent of these changes, you may need to re-run > MERT, re-do smoothing etc. > > Miles > > 2008/9/24 Panos <[EMAIL PROTECTED]>: >> Hello all >> >> >> >> Ok, the subject of my message covers 2 different topics so let's start > with >> the second one: Is it possible to enrich the lm and/or the phrase table? I >> mean, once we train the system with a specific corpus, is it possible to >> enrich it after we have gathered more data without retraining from > scratch? >> >> >> >> As for your suggestion Hieu, indeed this is what I want. I tampered a lot >> with the scripts in the web project, especially the translate.cgi, and I > am >> getting somewhere. However, my programming skills are still immature so I >> can't get the result I want. After I run daemon.pl, I can connect to my >> server with netcat, feed the sentence to moses, and get the output to my >> local machine. Unfortunately, the tokenizer and the detokenizer do not get >> invoked from inside the translate.cgi, so the translation I get is not > clean >> (it includes patterns like |0-9| between words/phrases). I know it should > be >> fairly easy for someone with perl knowledge to modify and strip down the >> translate.cgi file so it can be used with simple input or simple text > files >> instead of web pages. Then again, I think the only files needed to run a >> moses daemon for simple input/ouput are the translate.cgi, daemon.pl, and >> the lib folder with the files RemoteProcess.pm and Subprocess.pm. >> >> >> >> Any ideas and thoughts are welcome. >> >> >> >> Thank you. >> >> >> >> Panagiotis >> >> >> >> >> >> ________________________________ >> >> From: Hieu Hoang [mailto:[EMAIL PROTECTED] >> Sent: Friday, September 19, 2008 2:33 AM >> To: 'Panagiotis Kanavos'; [email protected] >> Subject: RE: [Moses-support] keep lm and phrase table in memory >> >> >> >> the >> >> /web >> >> subproject in the moses source code should give you a good example. The >> subproject creates a persistent process to translate incoming web queries. >> >> >> >> the alternative is to write a c++ program which calls the decoder > directly. >> The decoder is implemented as a library to enable you to do just that. >> >> >> >> Hieu Hoang >> >> www.hoang.co.uk/hieu >> >> >> >> >> >> ________________________________ >> >> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] >> On Behalf Of Panagiotis Kanavos >> Sent: 18 September 2008 22:33 >> To: [email protected]; Marcello Federico >> Subject: Re: [Moses-support] keep lm and phrase table in memory >> >> Thanks for your reply. I installed these programs and (partially) > connected >> the 2 systems. However, I can't find much documentation so I can use them >> properly, that is send the input from the local machine to moses in the >> server and then have it back translated while moses waits for the next >> sentence. >> >> On the local machine, I used sth like >> hose ServerName port_number --in sometext . >> >> and on the server I used sth like >> faucet port_number --out --verbose moses -f /path/to/moses.ini -i >> >> When I ran the command from the local machine, moses on the server indeed >> started loading the tables in memory but I didn't receive any output, > either >> on the server or on the local machine. I know I miss a lot of things so if >> you, or anyone else, know how to use these programs I would appreciate > your >> advice. >> >> Thank you. >> >> Panagiotis >> >> ________________________________ >> >>> From: [EMAIL PROTECTED] >>> To: [EMAIL PROTECTED]; [email protected] >>> Date: Thu, 18 Sep 2008 14:15:30 +0200 >>> Subject: RE: [Moses-support] keep lm and phrase table in memory >>> >>> Hi, >>> >>> Moses can be read text line-by-line from stdin and output translation to >>> stdout. Hence, >>> it should be very simple to set-up a client/server architecture through >>> the hose/faucet >>> linux commands. >>> >>> -- Marcello >>> >>> __________ >>> From: [EMAIL PROTECTED] [EMAIL PROTECTED] On >>> Behalf Of Panos [EMAIL PROTECTED] >>> Sent: Thursday, September 18, 2008 2:04 PM >>> To: [email protected] >>> Subject: [Moses-support] keep lm and phrase table in memory >>> >>> Hello, >>> >>> Is there a way to keep the language model and the phrase table in memory >>> without running the decoder? I am trying to connect moses to an external >>> application so that I can feed sentences for translation whenever I want. >>> However, having to load the lm and the phrase table into memory for every >>> single sentence I send is very time consuming. Moses runs on a Ubuntu > Hardy >>> 64bit server with 8GB of memory. >>> >>> Thank you. >>> >>> Panagiotis Kanavos >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
