Dear all, we are happy to invite you to participate in the Shared Task on Quality Estimation at WMT'24. The details of the task can be found at: https://www2.statmt.org/wmt24/qe-task.html
New this year: * We introduce a new language pair (zero-shot): English-Spanish * Continuing from the previous edition, we will also analyse the robustness of submitted QE systems to a set of different phenomena which will span from hallucinations and biases to localized errors, which can significantly impact real-world applications. * We also introduce a new task, seeking not only to detect but also to correct errors: Quality-aware Automatic Post-Editing! We invite participants to submit systems capable of automatically generating QE predictions for machine-translated text and the corresponding output corrections. 2024 QE Tasks: Task 1 -- Sentence-level quality estimation This task follows the same format as last year but with fresh test sets and a new language pair: English-Spanish. We will test the following language pairs: * English to German (MQM) * English to Spanish (MQM) * English to Hindi (MQM & DA) * English to Gujarati (DA) * English to Telugu (DA) * English to Tamil (DA) More details: https://www2.statmt.org/wmt24/qe-subtask1.html Task 2 -- Fine-grained error span detection Sequence labelling task: predict the error spans in each translation and the associated error severity: Major or Minor. We will test the following language pairs: * English to German (MQM) * English to Spanish (MQM) * English to Hindi (MQM) More details: https://www2.statmt.org/wmt24/qe-subtask2.html Task 3 -- Quality-aware Automatic Post-editing We expect submissions of post edits correcting detected error spans of the original translation. Although the task is focused on quality-informed APE, we also allow participants to submit APE output without QE predictions to understand the impact of their QE system. Submissions w/o QE predictions will also be considered official. We will test the following language pairs: * English to Hindi * English to Tamil More details: https://www2.statmt.org/wmt24/qe-subtask3.html Important dates: 1. Test sets will be released on July 15th. 2. Participants can submit their systems by July 23rd on codalab. 3. System paper submissions are due by 20th August [aligned with WMT deadlines]. Note: Like last year, we aligned with the General MT and Metrics shared tasks to facilitate cross-submission on the common language pairs: English-German, English-Spanish, and English-Hindi (MQM). We look forward to your submissions and feel free to contact us if you have any more questions! Best wishes, on behalf of the organisers.
