Dear Liam, There were several factors that led us to perform the manual evaluation only on the agent side (English-German). The customer source text was originally in English and was translated into German. We realised that showing the noisy English text to the evaluators as the reference could be misleading for them impacting their evaluations. Moreover, in this setting it is usually fine for the agent to have a gist of the message; however there is extra responsibility in translating the agent since it has the company brand with it. Therefore, we preferred to focus more on evaluating the agent translations more rigorously than to spend resources in evaluating the customer.
Hope this answers your question. Best, Amin > On 8 Oct 2020, at 12:35, LIANG DING <[email protected]> wrote: > > Dear Dr. Amin Farajian, > > Thank you for the effort to organize the manual evaluations of WMT Chat > Translations~ > > I have a question, why is there only English-German in this manual evaluation > result, but no opposite direction, i.e., German-English? > > Thank you again for your efforts and look forward to your Reply > > Liam > > Amin Farajian <[email protected] <mailto:[email protected]>> > 于2020年10月8日周四 下午6:40写道： > Dear Participants, > > We would like to inform you that the results of the manual document-level > direct assessments of all the primary submissions of the Chat Translation > shared task at WMT 2020 are now available on the website > http://www.statmt.org/wmt20/chat-task.html > <http://www.statmt.org/wmt20/chat-task.html>. > Moreover, the reference translations of the test set are also available on > the website. > > Thank you very much, > Amin > > -- > You received this message because you are subscribed to the Google Groups > "Workshop on Statistical Machine Translation" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/wmt-tasks/8ED074F8-D847-4516-9473-52924C05645D%40gmail.com > > <https://groups.google.com/d/msgid/wmt-tasks/8ED074F8-D847-4516-9473-52924C05645D%40gmail.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "Workshop on Statistical Machine Translation" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/wmt-tasks/CAHZN6XBuPc8iq4Fh%3DG3PSQPAdQRxawYEC9aA%3DO1Uszxv0DSAvA%40mail.gmail.com > > <https://groups.google.com/d/msgid/wmt-tasks/CAHZN6XBuPc8iq4Fh%3DG3PSQPAdQRxawYEC9aA%3DO1Uszxv0DSAvA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
