Hi Taylor,
If you are interested in playing datasets with human annotations for
translation quality (more specifically, 've released some time ago 2
types of datasets:
- 16,000 sentences, their reference translations, their machine
translations as produced by 4 SMT systems, and their scores in a {1-4}
scale for post-editing effort as given professional translators:
http://pers-www.wlv.ac.uk/~in1316/resources/ce_dataset.rar
- 1,000 English-Spanish and 2,525 French-English source sentences and
their machine translations, along their human post-edited version, 1-4
quality score, post-editing time, and HTER score:
http://pers-www.wlv.ac.uk/~in1316/resources/datasets_ce_eamt.tar.gz
The papers describing the datasets are (respectively):
- http://clg.wlv.ac.uk/papers/Specia_LREC2010.pdf
- http://clg.wlv.ac.uk/papers/EAMT-2011-Specia.pdf
Best,
Lucia
> Date: Mon, 19 Sep 2011 09:34:12 -0400
> From: Taylor Rose <[email protected]>
> Subject: Re: [Moses-support] Translation "Goodness"
> To: [email protected]
> Message-ID: <1316439252.17789.41.camel@Moses>
> Content-Type: text/plain; charset="UTF-8"
>
> Nguyen,
>
> Thanks again for the information. The system you describe is a great
> idea. I will post updates to the mailing list with any problems or
> successes I have.
>
> --
> Taylor Rose
> Machine Translation Intern
> Language Intelligence
>
>
> On Fri, 2011-09-16 at 23:53 -0400, Nguyen Bach wrote:
>> Hi Taylor and all,
>>
>> I am the first author of the "Goodness" paper and I would love to make
>> everything open source.
>> However, this work was done during my internship at IBM so everything
>> belongs to IBM.
>>
>> In order to replicate the work to some degrees, I suggest you use NIST
>> MT test sets and CRF++.
>> Steps can be
>> 1. Use your MT engine translate test sets.
>> 2. Use a TER aligner, for example TERp, to align your MT output with
>> translation references.
>> 3. Words without TER errors can be label as *Good* and others with TER
>> errors will be labeled *Bad*.
>> 4. Use CRF++, or any other ML toolkit, to train a binary classifier
>> with the features in the paper.
>> 5. Goodness score of a sentence can be computed by the sum of the
>> marginal probability of *Good* labels normalize by sentence length.
>>
>> I hope this suggestion will be helpful for you.
>>
>> Cheers,
>> Nguyen
>>
>> On 9/15/2011 1:52 PM, Barry Haddow wrote:
>> > Hi Taylor
>> >
>> > If I remember rightly, this paper made use of about 20-30k post-edited
>> > sentences which are unlikely to be released. So there is no way to
>> > replicate
>> > this work.
>> >
>> > Confidence estimation is an active research area in MT, but I don't think
>> > that
>> > there are any really good answers yet. Check out the last couple of years'
>> > ACL
>> > and EMNLP, as well as WMT, to see what's going on
>> > (http://www.aclweb.org/anthology-new/)
>> >
>> > cheers - Barry
>> >
>> > On Thursday 15 September 2011 18:26:22 Taylor Rose wrote:
>> >> Hey all,
>> >>
>> >> I've been researching how to judge the quality of a machine translation.
>> >> I found this article about judging the "goodness" of translations. This
>> >> is *exactly* what I've been trying to do. Does anyone know if their are
>> >> implementations of their algorithm available? It would take me a
>> >> substantial amount of time to try and replicate their process and even
>> >> then I do not have the corpus assets nor the processing power they had.
>> >>
>> >> Also, does anyone know of other existing systems that can accurately
>> >> compute the quality of translation without the need of an immense server
>> >> farm?
>> >>
>> >> Thanks,
>> >>
>>
>> _______________________________________________
>> Moses-support mailing list
>> [email protected]
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 19 Sep 2011 15:57:38 +0200
> From: Felipe S?nchez Mart?nez <[email protected]>
> Subject: Re: [Moses-support] Translation "Goodness"
> To: undisclosed-recipients:;
> Cc: [email protected]
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
>
> Hi,
>
> You may also be interested in this paper by Lucia Specia
> http://www.mt-archive.info/EAMT-2011-Specia.pdf
>
> Cheers
> --
> Felipe
>
> El 19/09/11 15:34, Taylor Rose escribi?:
>> Nguyen,
>>
>> Thanks again for the information. The system you describe is a great
>> idea. I will post updates to the mailing list with any problems or
>> successes I have.
>>
>
> --
> Felipe S?nchez Mart?nez
> Dep. de Llenguatges i Sistemes Inform?tics
> Universitat d'Alacant, E-03071 Alacant (Spain)
> Tel.: +34 965 903 400, ext: 2966 Fax: +34 965 909 326
> http://www.dlsi.ua.es/~fsanchez
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 19 Sep 2011 10:27:43 -0400
> From: Taylor Rose <[email protected]>
> Subject: Re: [Moses-support] Translation "Goodness"
> To: [email protected]
> Message-ID: <1316442463.19179.3.camel@Moses>
> Content-Type: text/plain; charset="UTF-8"
>
> Felipe,
>
> Wow that paper is great. Most of the other papers I've read don't
> provide their training data and say things like "we used a set of 40
> features" without stating what features they used. That was really
> frustrating for me since I am primarily a programmer and many linguistic
> techniques are not common sense to me yet.
>
> Thanks,
> --
> Taylor Rose
> Machine Translation Intern
> Language Intelligence
> IRC: Handle: trose
> Server: freenode
>
>
> On Mon, 2011-09-19 at 15:57 +0200, Felipe S?nchez Mart?nez wrote:
>> Hi,
>>
>> You may also be interested in this paper by Lucia Specia
>> http://www.mt-archive.info/EAMT-2011-Specia.pdf
>>
>> Cheers
>> --
>> Felipe
>>
>> El 19/09/11 15:34, Taylor Rose escribi?:
>> > Nguyen,
>> >
>> > Thanks again for the information. The system you describe is a great
>> > idea. I will post updates to the mailing list with any problems or
>> > successes I have.
>> >
>>
>
>
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 19 Sep 2011 11:15:14 -0400
> From: Nguyen Bach <[email protected]>
> Subject: Re: [Moses-support] Translation "Goodness"
> To: Philipp Koehn <[email protected]>
> Cc: [email protected]
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Phi,
>
> I haven't done this experiment.
>
> Nguyen
>
> On 9/18/2011 6:40 PM, Philipp Koehn wrote:
>> Hi,
>>
>> do you have any quantitative results on using post-edited texts vs.
>> parallel corpora, in terms of quality of the goodness measure?
>>
>> -phi
>>
>> On Sat, Sep 17, 2011 at 4:53 AM, Nguyen Bach<[email protected]> wrote:
>>> Hi Taylor and all,
>>>
>>> I am the first author of the "Goodness" paper and I would love to make
>>> everything open source.
>>> However, this work was done during my internship at IBM so everything
>>> belongs to IBM.
>>>
>>> In order to replicate the work to some degrees, I suggest you use NIST
>>> MT test sets and CRF++.
>>> Steps can be
>>> 1. Use your MT engine translate test sets.
>>> 2. Use a TER aligner, for example TERp, to align your MT output with
>>> translation references.
>>> 3. Words without TER errors can be label as *Good* and others with TER
>>> errors will be labeled *Bad*.
>>> 4. Use CRF++, or any other ML toolkit, to train a binary classifier
>>> with the features in the paper.
>>> 5. Goodness score of a sentence can be computed by the sum of the
>>> marginal probability of *Good* labels normalize by sentence length.
>>>
>>> I hope this suggestion will be helpful for you.
>>>
>>> Cheers,
>>> Nguyen
>>>
>>> On 9/15/2011 1:52 PM, Barry Haddow wrote:
>>>> Hi Taylor
>>>>
>>>> If I remember rightly, this paper made use of about 20-30k post-edited
>>>> sentences which are unlikely to be released. So there is no way to
>>>> replicate
>>>> this work.
>>>>
>>>> Confidence estimation is an active research area in MT, but I don't think
>>>> that
>>>> there are any really good answers yet. Check out the last couple of years'
>>>> ACL
>>>> and EMNLP, as well as WMT, to see what's going on
>>>> (http://www.aclweb.org/anthology-new/)
>>>>
>>>> cheers - Barry
>>>>
>>>> On Thursday 15 September 2011 18:26:22 Taylor Rose wrote:
>>>>> Hey all,
>>>>>
>>>>> I've been researching how to judge the quality of a machine translation.
>>>>> I found this article about judging the "goodness" of translations. This
>>>>> is *exactly* what I've been trying to do. Does anyone know if their are
>>>>> implementations of their algorithm available? It would take me a
>>>>> substantial amount of time to try and replicate their process and even
>>>>> then I do not have the corpus assets nor the processing power they had.
>>>>>
>>>>> Also, does anyone know of other existing systems that can accurately
>>>>> compute the quality of translation without the need of an immense server
>>>>> farm?
>>>>>
>>>>> Thanks,
>>>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> [email protected]
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 20 Sep 2011 11:43:10 +0430
> From: zeinab vakil <[email protected]>
> Subject: [Moses-support] I need to a graph including all hypotheses
> To: [email protected]
> Message-ID:
> <CAMMrkxaupKW=7QMz74iN=eeorpw3hchafr7nqxya4piczuc...@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> hello,
>
> Moses give best hypothesis for one sentence, but I need to a graph including
> all possible paths (all hypotheses) after pruning. I know that moses product
> such graph, but I don't know that how I can access it. Please guide me.
>
> vakil
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.mit.edu/mailman/private/moses-support/attachments/20110920/5aaa585d/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> End of Moses-support Digest, Vol 59, Issue 32
> *********************************************
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
