Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
oh right. The email about it is here http://article.gmane.org/gmane.comp.nlp.moses.user/5882/ Use the moses library but not the moses command line. Also built a C-based wrapper for the library - mainly to encourage people to develop gui in other languages, Java, C#, VB etc. It doesn't do lattice output though, as Holger wanted. On 06/06/2012 03:42, Lane Schwartz wrote: No, you weren't talking about the lattice output - you were talking about the moses library. On Tue, Jun 5, 2012 at 7:59 PM, Hieu Hoangfishandfrol...@gmail.com wrote: Wasn't me. I don't know much about the lattice output Hieu Sent from my flying horse On 5 Jun 2012, at 09:07 PM, Lane Schwartzdowob...@gmail.com wrote: I think Hieu mentioned recently that there is a Moses library that gets compiled, with an API that could be called. I've never used it, though. On Tue, Jun 5, 2012 at 3:49 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: On 06/05/2012 06:45 PM, Philipp Koehn wrote: Hi, An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? When using the switch -osgx FILE, a detailed score breakdown is provided with each line. I added this to the documentation: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc11 Hi, is there code somewhere to load such a graph into memory creating a suitable data structure ? Eventually functions to recalculate the global score given a set of feature weights and extracting the new best solution ? Once, I've that beast in memory it is pretty easy to rescore the LM probabilities with the CSLM. This code be also useful to do lattice MBR (independently from Moses) or lattice mert, and all kind of multi-pass decoding... Holger -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
On 06/04/2012 01:50 PM, Lane Schwartz wrote: Marcello, A GPL project can definitely use components from an LGPL project. So in the worst case, integration of the two could be done, distributing the whole combined work as a modification of CSLM. Not that I'm proposing we do that. In any case, what I meant by integration was doing the coding so that CSLM could be optionally compiled into Moses and used at runtime, just like SRILM, IRSTLM, and RandLM. Hello, it would be definitely interesting to be able to use the CSLM like any other LM during decoding. We could do this by just calling the corresponding functions when an LM probability is needed, but this risks to be quite inefficient. The code contains mechanisms to collect similar LM probability requests in order to minimize the number of forward passes of the neural network. In particular, different LM probability requests for the same word context should go together. It seems to me that this is also used in KenLM. The integration of the CSLM could also benefit from the work on the LM server. I guess that the LM requests are not sent individually to the server, but in larger bunchs. An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? best, Holger ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
On 06/05/2012 06:45 PM, Philipp Koehn wrote: Hi, An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? When using the switch -osgx FILE, a detailed score breakdown is provided with each line. I added this to the documentation: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc11 Hi, is there code somewhere to load such a graph into memory creating a suitable data structure ? Eventually functions to recalculate the global score given a set of feature weights and extracting the new best solution ? Once, I've that beast in memory it is pretty easy to rescore the LM probabilities with the CSLM. This code be also useful to do lattice MBR (independently from Moses) or lattice mert, and all kind of multi-pass decoding... Holger ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
I think Hieu mentioned recently that there is a Moses library that gets compiled, with an API that could be called. I've never used it, though. On Tue, Jun 5, 2012 at 3:49 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: On 06/05/2012 06:45 PM, Philipp Koehn wrote: Hi, An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? When using the switch -osgx FILE, a detailed score breakdown is provided with each line. I added this to the documentation: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc11 Hi, is there code somewhere to load such a graph into memory creating a suitable data structure ? Eventually functions to recalculate the global score given a set of feature weights and extracting the new best solution ? Once, I've that beast in memory it is pretty easy to rescore the LM probabilities with the CSLM. This code be also useful to do lattice MBR (independently from Moses) or lattice mert, and all kind of multi-pass decoding... Holger -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
Wasn't me. I don't know much about the lattice output Hieu Sent from my flying horse On 5 Jun 2012, at 09:07 PM, Lane Schwartz dowob...@gmail.com wrote: I think Hieu mentioned recently that there is a Moses library that gets compiled, with an API that could be called. I've never used it, though. On Tue, Jun 5, 2012 at 3:49 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: On 06/05/2012 06:45 PM, Philipp Koehn wrote: Hi, An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? When using the switch -osgx FILE, a detailed score breakdown is provided with each line. I added this to the documentation: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc11 Hi, is there code somewhere to load such a graph into memory creating a suitable data structure ? Eventually functions to recalculate the global score given a set of feature weights and extracting the new best solution ? Once, I've that beast in memory it is pretty easy to rescore the LM probabilities with the CSLM. This code be also useful to do lattice MBR (independently from Moses) or lattice mert, and all kind of multi-pass decoding... Holger -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
No, you weren't talking about the lattice output - you were talking about the moses library. On Tue, Jun 5, 2012 at 7:59 PM, Hieu Hoang fishandfrol...@gmail.com wrote: Wasn't me. I don't know much about the lattice output Hieu Sent from my flying horse On 5 Jun 2012, at 09:07 PM, Lane Schwartz dowob...@gmail.com wrote: I think Hieu mentioned recently that there is a Moses library that gets compiled, with an API that could be called. I've never used it, though. On Tue, Jun 5, 2012 at 3:49 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: On 06/05/2012 06:45 PM, Philipp Koehn wrote: Hi, An intermediate step could be to use the CSLM to rescore lattices which are likely to be a much richer dump of the search space than n-best lists. Can Moses create lattices which include all the (14) feature function scores ? When using the switch -osgx FILE, a detailed score breakdown is provided with each line. I added this to the documentation: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc11 Hi, is there code somewhere to load such a graph into memory creating a suitable data structure ? Eventually functions to recalculate the global score given a set of feature weights and extracting the new best solution ? Once, I've that beast in memory it is pretty easy to rescore the LM probabilities with the CSLM. This code be also useful to do lattice MBR (independently from Moses) or lattice mert, and all kind of multi-pass decoding... Holger -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
Marcello, A GPL project can definitely use components from an LGPL project. So in the worst case, integration of the two could be done, distributing the whole combined work as a modification of CSLM. Not that I'm proposing we do that. In any case, what I meant by integration was doing the coding so that CSLM could be optionally compiled into Moses and used at runtime, just like SRILM, IRSTLM, and RandLM. The situation with CSLM would be identical to RandLM, which is also GPL. Any code that's actually included with Moses, specifically a new LM wrapper in the moses/src/LM directory, should ideally be LGPL. But I would argue that there is no licensing problem with setting up the code so that a user can download Moses, separately download another third-party software distributed under a non-LGPL license, and the then compiling Moses with that third-party LM library. This is exactly what we already do with SRILM, RandLM, and ModelBlocks (the parser library that the syntactic LM uses). SRILM is distributed under a custom non-profit community license, RandLM is GPLv2, and ModelBlocks is GPLv3. Cheers, Lane On Mon, Jun 4, 2012 at 1:36 AM, Marcello Federico feder...@fbk.eu wrote: I suppose that an integration is not compatible with the current license of CSLM. GPL cannot be integrated into LGPL. Please, correct me if I'm wrong. Cheers, Marcello --- Short from my mobile phone On 04/giu/2012, at 06:12, Lane Schwartz dowob...@gmail.com wrote: Excellent! Thank you for releasing this, Holger! I know you had mentioned that you'd like to get this integrated into the decoder. Has anyone from your group been able to work on that? Cheers, Lane On Sun, Jun 3, 2012 at 7:13 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: I'm happy to announce the availability of a new version of the continuous space language model (CSLM) toolkit. Continuous space methods we first introduced by Yoshua Bengio in 2001 [1]. The basic idea of this approach is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown events can be expected. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still a n-gram approach, but the LM probabilities are interpolated for any possible context of length n-1 instead of backing-off to shorter contexts. CSLM were initially used in large vocabulary speech recognition systems and more recently in statistical machine translation. Improvements in the perplexity between 10 and 20% relative were reported for many languages and tasks. This version of the CSLM toolkit is a major update of the first release. The new features include: - full support for short-lists during training and inference. By these means, the CSLM can be applied to tasks with large vocabularies. - very efficient n-best list rescoring. - support of graphical extension cards (GPU) from Nvidia. This speeds up training by a factor of four with respect to a high-end server with two CPUs. We successfully trained CSLMs on large tasks like NIST OpenMT'12. Training on one billion words takes less than 24 hours. In our experiments, the CSLM achieves improvements in the BLEU score of up to two points with respect to a large unpruned back-off LM. A detailed description of the approach can be found in the following publications: [1] Yoshua Bengio and Rejean Ducharme. A neural probabilistic language model. In NIPS, vol 13, pages 932--938, 2001. [2] Holger Schwenk, Continuous Space Language Models; in Computer Speech and Language, volume 21, pages 492-518, 2007. [3] Holger Schwenk, Continuous Space Language Models For Statistical Machine Translation; The Prague Bulletin of Mathematical Linguistics, number 83, pages 137-146, 2010. [4] Holger Schwenk, Anthony Rousseau and Mohammed Attik; Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation, in NAACL workshop on the Future of Language Modeling, June 2012. The software is available at http://www-lium.univ-lemans.fr/cslm/. It is distributed under GPL v3. Comments, bug reports, requests for extensions and contributions are welcome. enjoy, Holger Schwenk LIUM University of Le Mans holger.schw...@lium.univ-lemans.fr ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
Excellent! Thank you for releasing this, Holger! I know you had mentioned that you'd like to get this integrated into the decoder. Has anyone from your group been able to work on that? Cheers, Lane On Sun, Jun 3, 2012 at 7:13 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: I'm happy to announce the availability of a new version of the continuous space language model (CSLM) toolkit. Continuous space methods we first introduced by Yoshua Bengio in 2001 [1]. The basic idea of this approach is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown events can be expected. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still a n-gram approach, but the LM probabilities are interpolated for any possible context of length n-1 instead of backing-off to shorter contexts. CSLM were initially used in large vocabulary speech recognition systems and more recently in statistical machine translation. Improvements in the perplexity between 10 and 20% relative were reported for many languages and tasks. This version of the CSLM toolkit is a major update of the first release. The new features include: - full support for short-lists during training and inference. By these means, the CSLM can be applied to tasks with large vocabularies. - very efficient n-best list rescoring. - support of graphical extension cards (GPU) from Nvidia. This speeds up training by a factor of four with respect to a high-end server with two CPUs. We successfully trained CSLMs on large tasks like NIST OpenMT'12. Training on one billion words takes less than 24 hours. In our experiments, the CSLM achieves improvements in the BLEU score of up to two points with respect to a large unpruned back-off LM. A detailed description of the approach can be found in the following publications: [1] Yoshua Bengio and Rejean Ducharme. A neural probabilistic language model. In NIPS, vol 13, pages 932--938, 2001. [2] Holger Schwenk, Continuous Space Language Models; in Computer Speech and Language, volume 21, pages 492-518, 2007. [3] Holger Schwenk, Continuous Space Language Models For Statistical Machine Translation; The Prague Bulletin of Mathematical Linguistics, number 83, pages 137-146, 2010. [4] Holger Schwenk, Anthony Rousseau and Mohammed Attik; Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation, in NAACL workshop on the Future of Language Modeling, June 2012. The software is available at http://www-lium.univ-lemans.fr/cslm/. It is distributed under GPL v3. Comments, bug reports, requests for extensions and contributions are welcome. enjoy, Holger Schwenk LIUM University of Le Mans holger.schw...@lium.univ-lemans.fr ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] New major release of the continuous space LM toolkit for SMT
I suppose that an integration is not compatible with the current license of CSLM. GPL cannot be integrated into LGPL. Please, correct me if I'm wrong. Cheers, Marcello --- Short from my mobile phone On 04/giu/2012, at 06:12, Lane Schwartz dowob...@gmail.com wrote: Excellent! Thank you for releasing this, Holger! I know you had mentioned that you'd like to get this integrated into the decoder. Has anyone from your group been able to work on that? Cheers, Lane On Sun, Jun 3, 2012 at 7:13 PM, Holger Schwenk holger.schw...@lium.univ-lemans.fr wrote: I'm happy to announce the availability of a new version of the continuous space language model (CSLM) toolkit. Continuous space methods we first introduced by Yoshua Bengio in 2001 [1]. The basic idea of this approach is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown events can be expected. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still a n-gram approach, but the LM probabilities are interpolated for any possible context of length n-1 instead of backing-off to shorter contexts. CSLM were initially used in large vocabulary speech recognition systems and more recently in statistical machine translation. Improvements in the perplexity between 10 and 20% relative were reported for many languages and tasks. This version of the CSLM toolkit is a major update of the first release. The new features include: - full support for short-lists during training and inference. By these means, the CSLM can be applied to tasks with large vocabularies. - very efficient n-best list rescoring. - support of graphical extension cards (GPU) from Nvidia. This speeds up training by a factor of four with respect to a high-end server with two CPUs. We successfully trained CSLMs on large tasks like NIST OpenMT'12. Training on one billion words takes less than 24 hours. In our experiments, the CSLM achieves improvements in the BLEU score of up to two points with respect to a large unpruned back-off LM. A detailed description of the approach can be found in the following publications: [1] Yoshua Bengio and Rejean Ducharme. A neural probabilistic language model. In NIPS, vol 13, pages 932--938, 2001. [2] Holger Schwenk, Continuous Space Language Models; in Computer Speech and Language, volume 21, pages 492-518, 2007. [3] Holger Schwenk, Continuous Space Language Models For Statistical Machine Translation; The Prague Bulletin of Mathematical Linguistics, number 83, pages 137-146, 2010. [4] Holger Schwenk, Anthony Rousseau and Mohammed Attik; Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation, in NAACL workshop on the Future of Language Modeling, June 2012. The software is available at http://www-lium.univ-lemans.fr/cslm/. It is distributed under GPL v3. Comments, bug reports, requests for extensions and contributions are welcome. enjoy, Holger Schwenk LIUM University of Le Mans holger.schw...@lium.univ-lemans.fr ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, Time Enough For Love ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
