David Nickerson wrote: > OK - now I'm really confused by all this talk of stars and simulation > tools. Perhaps the key is to separate the model curation status from the > simulation tools - i.e., to have a distinct "Curation Level" field at > the top of the model page separate to the current Download options > section. I think it is very import to ensure there is a clear > distinction between model curation status and how good or bad particular > simulation tools are in regard to a specific model. > > I'm not sure that we want to get rid of the multiple stars for > simulation tools, as defined by > http://www.cellml.org/repository-info/info. If someone comes along and > wants to look into using a particular "uncurated" model, it would be a > good starting point to know which tools can or cannot at least load the > model even if it doesn't give the right answers. >
Sure, but you only need a binary system to describe that - it either loads or it doesn't, one star, either there or not. More stars means confusion. >> Okay, I have just talked with Peter. He has decided that the units >> checking of models is a core issue and needs to be dealt with ASAP. I agree. >> >> We have decided to reform the current representation of curation levels. >> Firstly, the three stars for each simulator will go, and be replaced by >> one star, which represents that the model loads in that simulator. The >> argument against this is that all models should load in all simulators. >> This is, however not the case, refer to my comment later. Until it is >> the case, it is worth having simulator specific stars. > > Just to point out that this would be much better if it was framed as a > proposal rather than an explanation of a decision already reached. > Otherwise there is little point seeking comments... Sorry, the decision was for reform, the rest is what we talked about and are points for discussion. > > I'm a bit worried that you seem to be saying that by default all models > will get one star and then that star has to be manually removed? seems > the wrong way round to me. Hmm, I'd be worried if I was saying that too :) If that's how it came across I didn't mean that. Let me put it this way "the three *possible* stars for each simulator will go, and be replaced by one *possible* star." Is that clearer? > >> Secondly, all models which do not produce the output given in the paper >> will lose their one star. This first 'curation' star will be given to >> models which give the correct output. The problem with this, is that it >> is often not possible to reproduce the figures in a lot of the papers. >> These figures are often made by varying the initial values of variables >> or by using log scales etc. which would require adding components to the >> model or changing it such that it no longer accurately represents the >> model described in the paper. As such, we need a better way of judging >> whether our models are giving the right output. Where possible, I think >> contacting the authors of the paper might help here. This would >> hopefully then lead on to the model getting a second star. > > There is nothing to say that the analysis of simulation results has to > be performed in the same tool. For example, results can be exported from > PCEnv and checked using gnuplot or even Excel...having said that, I am > having some good success generating such figures using the graphing > metadata (which I really should finish writing up soon!) and imports to > look at things like restitution curves and pH dependence...but I think > that is for the future, we want to focus on the curation in regard to > the current repository features. > >> The second star will be for, as Andre said, "I'd be happy with giving a >> CellML 1.0 model without >>> reactions a curation status of level 2 if and only if it runs in PCEnv >>> and JSim without error and gives identical results (within some >>> numerical tolerance) and passes Jonathan's validation tests without >>> error." (What do you mean identical results? Identical results between >> simulators or identical to the figures in the paper? If the latter, how >> do you quantify this?) This second star will in most cases require the >> author to fix the model so that it satisfies the validation tests, such >> as unit testing. This obviously presents a problem for the models that >> were made in the 60's and 70's. Most of these models are pretty seminal >> though, so hopefully we won't have too many problems with them. > > Now it gets confusing....this particular second star is for the curation > level, right? rather than a specific simulation tool? My suggestion was > just one way that could be used to be satisfied that a model passes the > requirements for level 2 curation without sitting down and manually > checking every variable and equation in a model. Okay, the idea is that: there are two sets of stars. One for curation, one for the simulator. For curation, there are three possible stars, as outlined here. For the simulator, there is one possible star, which is given if the model loads in that simulator. Identical results in > this sense means that if you export the results from different > simulation tools and plot them together that the curves all overlay - a > more rigorous check would be to look at RMS or some other error > measurement between the different simulation result sets. ok > >> The third star will be for models that have been independently peer >> reviewed by a domain specific expert. Peter does not anticipate that any >> of our models will fit this status for a while. > > yep - and we still need to agree on what level 3 curation really means, > but we can talk about that in your next thread... sure - I doubt we're going to get there in a hurry though. > >> Currently, I think the biggest issue is with getting models to produce >> the output that the original model produced. I think sometimes the best >> we'll be able to do is ensure that the model is mathematically >> equivalent to what is described by the authors. > > Sure, and then the model sits at level 1 curation until someone comes > along to fix it up. There is nothing wrong with that. apart from people coming along and thinking that because the models only get 1/3 they aren't curated well. If we're having problems with our system how are we going to expect other people to understand! :) > >> Please comment. > > ok :-) > >>> Until CellML 1.1 models are in the repository, all the models (without >>> reaction elements) at level two curation status should run with no error >>> in JSim and COR and PCEnv (and any other simulation tool that works). >> This is the ideal case, however I have noticed that there are >> differences in which models COR can run and which models PCEnv can run. >> Since I work in Linux, and COR doesn't run in Linux, I have to reboot my >> computer to check out a model in COR. Either that or I have to get on >> the computer next to me, transfer the appropriate files by email or USB >> stick, and work there. Currently that computer is being used by someone >> else who is doing cellular automata simulations that take hours at a >> time, and I don't want to interfere with his work. This is why many of >> the models have a star for PCEnv but not for COR. The only models that i >> know to work in COR are those that Penny has sent me, and those few that >> I've actually tried in COR. > > thats why I'm suggesting JSim as that runs fine under linux. I think the > issue that you are talking about is that PCEnv has support for the > largest range of MathML and the way equations can be expressed. So there > will be some cases where models that run in PCEnv won't run in other > simulation tools. For these cases there are sometimes some simple > changes that can be made to get the model running in COR or JSim. > However, if you can't get the model to load into JSim or Jonathan's > validation tool (PyCML), then units checking becomes an issue. You can > do it manually to curate models to level 2, but as I think you know that > is quite a tedious and error prone approach. But until the units have > been checked the model can not be given the two curation stars. > >>> And just a final note for the model status documentation. You seem to >>> add to previous documentation which in some cases starts to get a bit >>> misleading. For example, the jafri_rice_winslow_1998_version01_variant01 >>> model has probably not been validated by Penny? It would probably make >>> more sense to simply state the version/variant upon which the model >>> comes from and what the new status of the model is....if that makes sense. >> This is infact a file that has been curated by Penny - she sent it to me >> and I uploaded it. What is confusing is the version/variant issue. Is >> this version 01 of variant 01, or is it variant 01 of version 03? Until >> this is sorted out, there appears to be no 'right' way to do it. >> Granted, this is in fact an inconsistency in the way i've been doing it >> (the latter way,) so thanks for pointing this out. > > whoops, my bad! > > hopefully Tommy's design for moving the repository forward will help > resolve the whole versions of variants and variants of versions issue. > I think Matt needs to work on the naming convention for this to be cleared up. This is an issue that keeps coming up. > > Thanks, > David. > _______________________________________________ > cellml-discussion mailing list > [email protected] > http://www.cellml.org/mailman/listinfo/cellml-discussion _______________________________________________ cellml-discussion mailing list [email protected] http://www.cellml.org/mailman/listinfo/cellml-discussion
