Website: https://humeval.github.io/
Previous edition (at EACL 2021): https://humeval.github.io/2021/ Workshop overview The Second Workshop on Human Evaluation of NLP Systems (HumEval’22) invites the submission of long and short papers on substantial, original, and unpublished research on all aspects of human evaluation of NLP systems with a focus on NLP systems which produce language as output. We welcome work on any quality criteria relevant to NLP, on both intrinsic evaluation (which assesses systems and outputs directly) and extrinsic evaluation (which assesses systems and outputs indirectly in terms of its impact on an external task or system), on quantitative as well as qualitative methods, score-based (discrete or continuous scores) as well as annotation-based (marking, highlighting). Invited speakers Samira Shaikh, University of North Carolina and Ally Financial Inc. Markus Freitag, Google Research Important dates - 28 February 2022: Submission Deadline - 22 March 2022: Commitment deadline for ARR submissions with reviews - 26 March 2022: Notification of Acceptance - 10 April 2022: Camera-ready papers due - 27 May 2022: Workshop at ACL All deadlines are 23:59h UTC-12 Papers We invite papers on topics including, but not limited to, the following: - Experimental design and methods for human evaluations - Reproducibility of human evaluations - Approaches to validating human evaluation methods - Work on inter-evaluator and intra-evaluator agreement - Ethical considerations in human evaluation of computational systems - Quality assurance for human evaluation - Crowdsourcing for human evaluation - Issues in meta-evaluation of automatic metrics by correlation with human evaluations - Alternative forms of meta-evaluation and validation of human evaluations - Comparability of different human evaluations - Methods for assessing the quality and the reliability of human evaluations - Role of human evaluation in the context of Responsible and Accountable AI We welcome work from any subfield of NLP (and ML/AI more generally), with a particular focus on evaluation of systems that produce language as output. Submission Long papers Long papers must describe substantial, original, completed and unpublished work. Wherever appropriate, concrete evaluation and analysis should be included. Long papers may consist of up to eight (8) pages of content, plus unlimited pages of references. Final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers’ comments can be taken into account. Long papers will be presented orally or as posters as determined by the programme committee. Decisions as to which papers will be presented orally and which as posters will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between long papers presented orally and as posters. Short papers Short paper submissions must describe original and unpublished work. Short papers should have a point that can be made in a few pages. Examples of short papers are a focused contribution, a negative result, an opinion piece, an interesting application nugget, a small set of interesting results. Short papers may consist of up to four (4) pages of content, plus unlimited pages of references. Final versions of short papers will be given one additional page of content (up to 5 pages) so that reviewers’ comments can be taken into account. Short papers will be presented orally or as posters as determined by the programme committee. While short papers will be distinguished from long papers in the proceedings, there will be no distinction in the proceedings between short papers presented orally and as posters. Multiple submission policy HumEval22 allows multiple submissions. However, if a submission has already been, or is planned to be, submitted to another event, this must be clearly stated in the submission form. Submission procedure and templates Submission is electronic, through the OpenReview portal for the workshop: https://openreview.net/group?id=aclweb.org/ACL/2022/Workshop/HumEval with the deadline on 28 February 2022. Both long and short papers must be anonymised for double-blind reviewing, must follow the ACL Author Guidelines <https://www.aclweb.org/adminwiki/index.php?title=ACL_Author_Guidelines>, and must use the ACL 2022 templates available here https://aclrollingreview.org/cfp. The submitting author must have an OpenReview profile. Please ensure profiles are complete before submission. This tutorial <https://docs.google.com/presentation/d/1kJeoAfwbnFapUN0ySLSoOm11-2odz48DGS1DEzNs03k/edit#slide=id.gcfa2063058_0_0> from the ACL Rolling Review might be helpful. In addition to standard submissions, submissions which were previously submitted to ARR and reviewed there can be committed to the workshop (together with the reviews) not later than 22 March 2022. We suppose that all papers submitted to ARR until 15 February 2022 will be reviewed by 22 March, however we cannot guarantee because it depends on ARR editors and not on us. Optional Supplementary Materials: Appendices, Software and Data ARR encourages the submission of these supplementary materials to improve the reproducibility of results, and to enable authors to provide additional information that does not fit in the paper. Supplementary materials may include appendices, software or data. For example, pre processing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper can be put into appendices. However, if the pseudo-code or derivations or model specifications are an important part of the contribution, or if they are important for the reviewers to assess the technical correctness of the work, they should be a part of the main paper, and not appear in appendices. Reviewers are not required to consider material in appendices. Appendices should come after the references in the submitted pdf, but do not count towards the page limit. Software should be submitted as a single .tgz or .zip archive, and data as a separate single .tgz or .zip archive. Supplementary materials must be fully anonymized to preserve the two-way anonymized reviewing policy. Organisers Anya Belz, ADAPT Centre, Dublin City University, Ireland Maja Popović, ADAPT Centre, Dublin City University, Ireland Ehud Reiter, University of Aberdeen, UK Anastasia Shimorina, Orange, Lannion, France For questions and comments regarding the workshop please contact the organisers at [email protected]. Programme committee Eleftherios Avramidis, DFKI, Germany Sheila Castilho, ADAPT, Dublin City University, Ireland Sandipan Dandapat, Microsoft, India Ondrej Dušek, Charles University, Czechia Markus Freitag, Google, USA Albert Gatt, Malta University, Malta Behnam Hedayatnia, Amazon, USA David Howcroft, Heriot Watt University, UK Tom Kocmi, Microsoft, Germany Filip Klubička, ADAPT, Technological University of Dublin, Ireland Samuel Läubli, University of Zurich, Switzerland Chris van der Lee, Tilburg University, Netherlands Saad Mahamood, Trivago, Germany Nitika Mathur, University of Melbourne, Australia Margot Mieskes, UAS Darmstadt, Germany Emiel van Miltenburg, Tilburg University, Netherlands Mathias Mueller, University of Zurich, Switzerland Sergiu Nisioi, University of Bucharest, Romania Juri Opitz, University of Heidelberg, Germany Maike Paetzel-Prüsmann, University Potsdam, Germany Maxime Peyrard, EPFL, Switzerland Tim Polzehl, TU Berlin, Germany Martin Popel, UFAL, Czechia Verena Rieser, Heriot Watt University, UK Samira Shaikh, UNC, USA Wei Zhao, TU Darmstadt, Germany
_______________________________________________ Mt-list site list [email protected] https://lists.eamt.org/cgi-bin/mailman/listinfo/mt-list
