Hello world, the 3rd WMT shared task on quality estimation, which basically consists of submitting an MT metric that does not use the reference (only the source and hypothesis) is still open! The deadline and submission procedure are the same as for regular reference-using (boring) metrics; see Ondrej's links for the submissions to score and the document to describe your submission duplicated below. *Deadline: May 25th*.
Although "metrics without references" sounds like usual QE (predict quality using source and hypothesis), the key difference is that multiple MT systems produce the hypotheses that you score, so we expect this evaluation to move QE away from fitting models to a particular MT system and force them to generalize better. We especially encourage QE task 1 and 2 participants to apply their already trained models to English-Russian and English-German news translations from task 3 -- after all, if HTER/MQM do not correlate at least somewhat with human understanding of MT quality, then something is wrong. Standalone submissions to task 3 for any translation directions are of course also encouraged! So, submit a referenceless metric and show BLEU (and TER and CHRF and etc) that single-reference problems can be solved with no-reference! (fingers crossed). Ondrej's info on submitting metrics (both reference-free or reference-dependent) is below. Best wishes (and good luck with WMT papers), Mark ---------- Forwarded message --------- From: Ondrej Bojar <bo...@ufal.mff.cuni.cz> Date: Fri, May 10, 2019 at 2:07 AM Subject: WMT19 Metrics Task inputs ready To: wmt-tasks <wmt-ta...@googlegroups.com> Cc: qingsong ma <qingsong....@gmail.com>, jwei <j...@umass.edu>, Yvette Graham <graham.yve...@gmail.com> Dear all, WMT19 Metrics Task inputs, i.e. MT system outputs to score, are ready for you to process, see: http://www.statmt.org/wmt19/metrics-task.html#download As the top of the page says, the submission for your metrics papers is indeed *earlier* that we need the final scores from you. We expect you to update your scores for the camera-ready version. Please remember to sign up yourself to the shared spreadsheet so that we know that we need to expect data from you: https://docs.google.com/spreadsheets/d/1B-0grtoJoSOfGCb4xrlkZ7dpxraFqezGumSmE2rpldE/edit?usp=sharing Also please get in touch if your metric is able to score document-level translation quality. Looking forward to your submissions! Ondrej, on behalf WMT Metrics Task organizers
_______________________________________________ Mt-list site list Mt-list@eamt.org http://lists.eamt.org/mailman/listinfo/mt-list
