CALL FOR PARTICIPATION
Machine Translation Evaluation Workshop & Shared Task
http://romip.ru/mteval/
[email protected]
AIM & SCOPE
The aims of the workshop are (1) to develop a common testbed and
(2) to evaluate different machine translation approaches for
English-Russian pair. Translation will be evaluated on an unseen
test set using different machine translation evaluation methods.
The workshop is organized by the Russian Information Retrieval
Evaluation Seminar (ROMIP, http://romip.ru) in cooperation with
TAUS Labs (http://tauslabs.com/).
The workshop is open to all kinds of MT systems and technologies.
Experienced and early-stage researchers, as well as industrial
developers, are welcome to participate in the evaluation campaign
and the workshop.
The workshop will take place at the Dialog conference on
computational linguistics and intelligent technologies (
http://www.dialog-21.ru/dialog2013/).
DATA
A test dataset of about 150,000 sentences originally written in
English will be made available to the participants. The
participants will be requested to submit the whole dataset
translated into Russian.
The participants are free to use their own systems and any data to
complete the task. For example, participants can use the following
freely available resources:
- 1M sentences English-Russian parallel corpus released by Yandex;
- 119K English-Russian parallel corpus from the TAUS Data
Repository.
Organizers leave to participants' discretion to use or not to use
these data.
EVALUATION
The evaluation will be completed on about 1,000 sentences from the
test dataset. Human translators will translate these sentences to
ensure the gold standard quality. We will employ two types of
evaluation measures:
- automated metrics widely adopted by statistical machine
translation community;
- blind pairwise evaluation of systems' output performed by human
assessors.
We expect the participants to share organizational costs either by
taking part in human assessment or in monetary form.
IMPORTANT DATES
20 December 2012 - announcement & data samples
15 January 2013 - release of complete dataset
22 January 2013 - deadline for runs submission
22 February 2013 - announcement of evaluation results
20 March 2013 - deadline for report submission
29 May - 2 June 2013 - workshop @ Dialog conference
ORGANIZERS
Pavel Braslavski (Kontur Labs/Ural Federal University)
Maxim Khalilov (TAUS Labs)
Sergey Sharoff (University of Leeds)
CONTACTS
Inquiries and suggestions are welcome at [email protected]
(upon joining the mailing list at
https://groups.google.com/d/forum/MTeval )
_______________________________________________
Mt-list mailing list
[email protected]
http://mailhost.computing.dcu.ie/mailman/listinfo/mt-list
