ClimateCheck: Shared Task on Scientific Fact-Checking of Social Media Claims on
Climate Change
Hosted as part of the SDP 2025 Workshop
July 31 or August 1st, 2025 (tbc)
Vienna, Austria
(co-located with ACL 2025)
ClimateCheck Shared Task: https://sdproc.org/2025/climatecheck.html
SDP 2025 Workshop: https://sdproc.org/2025/index.html
Task Overview
Social media facilitates discussions on critical issues such as climate change,
but it also contributes to the rapid dissemination of misinformation, which
complicates efforts to maintain an informed public and create evidence-based
policies. In this shared task, we emphasise the need to link public discourse
to peer-reviewed scholarly articles by gathering claims from social media about
climate change (both real-life and automatically generated ones) as well as a
corpus of about 400K abstracts of publications from the climate sciences
domains. The participants will be asked to retrieve relevant abstracts for each
claim (subtask I) and classify the relation between the claim and abstract as
‘supports’, ‘refutes’, or ‘not enough information’ (subtask II). The task will
be hosted on Codabench (link will be provided soon on the webpage).
Participants are allowed to take part either in subtask I only, or in both
subtasks.
Subtask I: Abstracts Retrieval
Task: given a claim from social media about climate change and a corpus of
abstracts, retrieve the top K most relevant abstracts.
Evaluation: MAP and B-Pref accounting for retrieving relevant abstracts and not
penalising unjudged documents.
Subtask II: Claim Verification
Task: given the claim-abstract pairs received from the previous subtask,
classify their relation as ‘support’, ‘refutes’, or ‘not enough information’.
Evaluation: F1 score based on judged documents from gold data; unjudged
documents will not be included in computing the score.
Important dates
Release of training data: April 1, 2025
Release of testing data: April 15, 2025
Deadline for system submissions: May 16, 2025
Paper submission deadline: May 23, 2025
Notification of acceptance: June 13, 2025
Camera-ready paper due: June 20, 2025
Workshop: July 31, 2025 or August 1, 2025 (TBA)
We encourage and invite participation from junior researchers and students from
diverse backgrounds. Participants are also encouraged to submit a paper
describing their systems to the SDP 2025 workshop.
Organisers
Raia Abu Ahmad (DFKI, Berlin, Germany)
Aida Usmanova (Leuphana University, Lüneburg, Germany)
Georg Rehm (DFKI, Berlin, Germany)
_______________________________________________
Corpora mailing list -- [email protected]
https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/
To unsubscribe send an email to [email protected]
