(apologies for multiple copies; this is not a MT task but related) *** Second Shared Task on Automatic Minuting *** *** AutoMin 2023 as INLG Challenge ***
Automatic summarization of meetings into meeting minutes (aka `minuting') would be an amazing tool. Superficially, large language models like BART, GPT* and friends may seem to have solved it already, but serious concerns still remain regarding completeness and factual correctness. We would like invite you to take part in AutoMin 2023, the second instance of the shared on automatic minuting, i.e. summarization of English and Czech meeting transcripts into meeting minutes and their evaluation. See the details on task options below. AutoMin 2023 is a registered INLG Challenge this year. *** Website *** https://ufal.github.io/automin-2023/ *** Tasks *** We invite you to take part in one or both main tasks (Task A and D), and optionally in two subsidiary tasks (Task B and C): - Main Task A: The main task consists of automatically creating minutes from multiparty meeting transcripts. The generated minute will be evaluated both via automatic and manual metrics. - Task B: Given a pair of meeting transcripts and minutes, the task is to identify whether the minutes belong to the transcript. During our data preparation from meetings on similar topics, we found that this task could be challenging given repetition and not straightforward correspondences between transcripts and minutes. - Task C: Given a pair of minutes, the task is to identify whether the two minutes belong to the same or different meetings. This sub-task is important as we want to uncover how minutes created by two different persons for the same meeting may differ in content and coverage. - Task D: Given a meeting transcript, a candidate minutes, and a set of one or more reference minutes, assign a score indicating the quality of the candidate. Task D will not be evaluated by a single criterion. All submissions to Task D will be evaluated in terms of Pearson correlation against all manual and all automatic evaluation scores available in AutoMin 2023, including other submissions to Task D. *** Registration *** All participants should register using the following link: https://forms.office.com/e/jdvsdtV5Tm Registration and participation is free of charge. Please get in touch in case you would be willing to contribute to manual evaluation of AutoMin submissions. *** Important Dates *** Feb 6, 2023 All Training and Dev Set Data Available Feb 20, 2023 Test Data Available Apr 15, 2023 System Submission Deadline May 3, 2023 Report Submission Deadline TBA Camera Ready Submission; in line with INLG September, 2023 AutoMin @ INLG 2023 *** Publication *** All teams are required to submit a brief technical report describing their method. Please submit using the INLG paper template (to be announced) All reports must be a minimum of 2 pages and a maximum of 4 pages excluding references. Reports must be written in English. The proceedings will be published in ACL Anthology. *** Organizers *** Tirthankar Ghosal, UFAL, Charles University, Czech Republic Muskaan Singh, UFAL, Charles University, Czech Republic Anja Nedoluzhko, UFAL, Charles University, Czech Republic Ondřej Bojar, UFAL, Charles University, Czech Republic *** Contact *** For inquiries, contact <[email protected]>. -- Ondrej Bojar (mailto:[email protected] / [email protected]) http://www.cuni.cz/~obo _______________________________________________ Moses-support mailing list [email protected] https://mailman.mit.edu/mailman/listinfo/moses-support
