Peter Thanks for your answer and the detailled how-to.
In the meatime, I used some of your classes (essentially the ruta.testing.evaluator ones) and build an standalone evaluator tool that takes both goldStandard/ run/ folders with UIMA xmi inside as argument, plus a list of annotations plus a method (partial, exact) and produce a detailled result. This works fine, however it takes about 2 minutes to analyse 300 xmi files. Is this amount of time sounds normal to you ? I find that way to long. If so, I would be glad to share the result because while "easy" to write, this kind of program should be packaged into UIMA core I guess. Le 20 mars 2018 à 11:41, Peter Klügl écrivait : > Hi, > > > I think there is no common default implementation for this, but everyone > has its own implementation. For the normal uses cases this can be > implemented easily. However, it gets complicated if you need fancy > features, e.g., comparing different levels of complex feature values. > > > Normally, you have something like: > > 1. CasReader for providing the expected gold annotations > > 2. AnnotationCopier for moving the annotations to a gold view > > 3. Annotators of your pipeline for created the annotations > > 4. AnnotationComparator for comparing the new annotation with the > annotations of the gold view > > 5. EvaluationWriter for aggregating and storing the evaluation result > > > I personally do not use the Ruta evaluation anymore since it does not > provide enough features and our evaluations are integrated in your maven > build as integration tests, thus no need for the Ruta GUI/Annotation > Testing View. > > > Best, > > > Peter > > > Am 17.03.2018 um 20:31 schrieb Nicolas Paris: > > Hello, > > > > The RUTA workbench Annotation Test is a great tool to evaluate the > > performances of a RUTA script based on a gold standard. > > > > Is there any existing tool to measure the performances based on one > > input/ouput xmi folders? > > > > I guess it is feasible to hack the RUTA workbench by running uimafit > > pipelines from ruta, but since my pipeline has many annotators engines, > > this looks complicated to do. > > > > Thanks, > > -- > Peter Klügl > R&D Text Mining/Machine Learning > > Averbis GmbH > Tennenbacher Str. 11 > 79106 Freiburg > Germany > > Fon: +49 761 708 394 0 > Fax: +49 761 708 394 10 > Email: peter.klu...@averbis.com > Web: https://averbis.com > > Headquarters: Freiburg im Breisgau > Register Court: Amtsgericht Freiburg im Breisgau, HRB 701080 > Managing Directors: Dr. med. Philipp Daumke, Dr. Kornél Markó >
