Dear colleagues, I am looking for volunteer annotators for a research study on the human evaluation of temporal question-answering, RAG, and GraphRAG systems.
The study investigates whether existing and newly developed automatic metrics can reliably evaluate answers that depend on temporal facts, retrieved evidence, and graph-based reasoning. Human judgments will serve as the reference against which these metrics are compared. Annotation task Participants will evaluate 20 system-generated answers. Depending on the sample, the judgments concern: - answer correctness; - temporal correctness; - whether the supplied evidence supports the answer; - whether citations are temporally appropriate; - whether graph evidence is sufficient; and - whether a system’s decision to answer or decline to answer is appropriate. All required questions, evidence, graph information, and reference material are provided in the annotation interface. External search and AI tools should not be used. Expected commitment - Guided tutorial: approximately 15 minutes - Main annotation task: approximately 60-80 minutes Total expected time: approximately 1.5 hours Progress is saved automatically, allowing the task to be paused and resumed A desktop or laptop computer is strongly recommended The dataset and interface are entirely in English. Participants should therefore be fluent English readers. Experience with NLP, information retrieval, knowledge graphs, question answering, RAG, or LLM evaluation is helpful but not required. No prior familiarity with this project is necessary. This is a voluntary and unpaid academic contribution. Access is distributed individually rather than through a public link. Each participant receives a private study URL, an annotation guide, and a pseudonymous participant ID. To participate, please contact me at: [ mailto:[email protected] | [email protected] ] with the subject: Temporal RAG annotation study Please feel free to forward this call to colleagues, researchers, students, or practitioners who may be interested. Best regards, Murad Mustafayev
_______________________________________________ Corpora mailing list -- [email protected] https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/ To unsubscribe send an email to [email protected]
