http://sakailab.com/fairweb1/
# Overview The FairWeb-1 task is a new English web search task that considers both relevance and group fairness. Each of our search topic seeks information about specific entity types: R (researher), M (movies), T (Twitter accounts), and Y (YouTube contents); accordingly, we have four topic types: R-topics (e.g., information retrieval researchers) M-topics (e.g., Daniel Craig 007 movies) T-topics (e.g., Twitter accounts that provide info on COVID) Y-topics (e.g., Coldplay covers on YouTube) For evaluating group fairness, we consider the following attribute sets containing either ordinal or nominal groups: R-topics: h-index (ordinal) AND gender (nominal) M-topics: #reviews on IMDb (ordinal) AND geographic region (nominal) T-topics: #followers (ordinal) Y-topics: #subscribers of the YouTube account (ordinal) For R- and M-topics, we will evaluate intersectional groupfairness. For example, for each R-topic, we want the search engine result page (SERP) to contain information about researchers with varying levels of h-index (not just high h-index people), AND with different genders (not just "men"). # Task input/ouput INPUT: - a search topic (R, M, T, or Y) - attribute set(s) and target distribution(s) OUTPUT: - a TREC-style run (a SERP for each topic) # Evaluation method We will use the Group Fairness and Relevance (GFR) framework: please visit our webpage for details. # Timeline October 7, 2022 Release of 1st CFP with sample topics and evaluation protocol December 16, 2022 Pilot relevance assessments for the sample topics and a few pilot runs released; topic set size determined Dec 19-Feb 28, 2023 Topic development March 1, 2023 Topics released; task registrations due May 12, 2023 Run submissions due May 15-July 31, 2023 Entity annotations; runs evaluated August 1, 2023 Evaluation results and draft overview released September 1, 2023 Draft participant papers due November 1, 2023 Camera ready papers due December 2023 NTCIR-17@NII, Tokyo, Japan # Inquiries: [email protected] # Organisers: Sijie Tao, Nuo Chen, Tetsuya Sakai (Waseda University, Japan) Zhumin Chu (Tsinghua University, P.R.C.) Nicola Ferro (University of Padua, Italy) Maria Maistro (University of Copenhagen, Denmark) Ian Soboroff (NIST, USA) Hiromi Arai (RIKEN AIP, Japan)
_______________________________________________ Corpora mailing list -- [email protected] https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/ To unsubscribe send an email to [email protected]
