Dear All, Okay, I have just talked with Peter. He has decided that the units checking of models is a core issue and needs to be dealt with ASAP. I agree.
We have decided to reform the current representation of curation levels. Firstly, the three stars for each simulator will go, and be replaced by one star, which represents that the model loads in that simulator. The argument against this is that all models should load in all simulators. This is, however not the case, refer to my comment later. Until it is the case, it is worth having simulator specific stars. Secondly, all models which do not produce the output given in the paper will lose their one star. This first 'curation' star will be given to models which give the correct output. The problem with this, is that it is often not possible to reproduce the figures in a lot of the papers. These figures are often made by varying the initial values of variables or by using log scales etc. which would require adding components to the model or changing it such that it no longer accurately represents the model described in the paper. As such, we need a better way of judging whether our models are giving the right output. Where possible, I think contacting the authors of the paper might help here. This would hopefully then lead on to the model getting a second star. The second star will be for, as Andre said, "I'd be happy with giving a CellML 1.0 model without > reactions a curation status of level 2 if and only if it runs in PCEnv > and JSim without error and gives identical results (within some > numerical tolerance) and passes Jonathan's validation tests without > error." (What do you mean identical results? Identical results between simulators or identical to the figures in the paper? If the latter, how do you quantify this?) This second star will in most cases require the author to fix the model so that it satisfies the validation tests, such as unit testing. This obviously presents a problem for the models that were made in the 60's and 70's. Most of these models are pretty seminal though, so hopefully we won't have too many problems with them. The third star will be for models that have been independently peer reviewed by a domain specific expert. Peter does not anticipate that any of our models will fit this status for a while. Currently, I think the biggest issue is with getting models to produce the output that the original model produced. I think sometimes the best we'll be able to do is ensure that the model is mathematically equivalent to what is described by the authors. Please comment. David Nickerson wrote: >> I haven't been using Jsim, I've mainly been using PCEnv. It is my >> understanding that I can't use Jsim yet, from what others have told me, >> but I haven't checked that out myself. > > I can't recall any reason why you couldn't use JSim - just like > Jonathan's validator it will give you errors for inconsistent units. > Then once the units are corrected you'll be able to run simulations and > make sure that the corrected units version still gives correct results. I'll look into it. > > Until CellML 1.1 models are in the repository, all the models (without > reaction elements) at level two curation status should run with no error > in JSim and COR and PCEnv (and any other simulation tool that works). This is the ideal case, however I have noticed that there are differences in which models COR can run and which models PCEnv can run. Since I work in Linux, and COR doesn't run in Linux, I have to reboot my computer to check out a model in COR. Either that or I have to get on the computer next to me, transfer the appropriate files by email or USB stick, and work there. Currently that computer is being used by someone else who is doing cellular automata simulations that take hours at a time, and I don't want to interfere with his work. This is why many of the models have a star for PCEnv but not for COR. The only models that i know to work in COR are those that Penny has sent me, and those few that I've actually tried in COR. > Unless someone has some examples of where JSim is failing? in which case > I'm sure the JSim guys would be keen to help rectify the problem. > >> Regarding the star system, I agree it is confusing. I'm just using it >> because it's there. If you have any ideas on a more robust way on >> presenting curation status to users, I'm all ears. >> >> I agree unit checks are really essential to properly validating a model, >> but often there doesn't seem to be a way of getting everything to check >> out. > > yet models are still being flagged as curated to level 2 curation - > which requires that all units are consistent. > >> For example, take this relatively simple model that I've just fixed. >> I've given it two stars because it nicely reproduces the figures in the >> paper it comes from. However, when I run Jonathan Cooper's validator on >> it, the following result comes up: >> >> >> "[EMAIL PROTECTED]:~/Desktop/tools/JC$ python validator.py >> ~/Desktop/goldbeter3.cellml >> Validating file /people/jlaw060/Desktop/goldbeter3.cellml against CellML 1.0 >> Error checking units: Operator plus requires its operands to have >> dimensionally equivalent units; dimensionless and first_order_rate differ >> Context: 1st expression in the 1st math element in component M >> XPath: /*[1]/*[4]/*[8]/*[1]/*[3]/*[2]/*[3] >> Error checking units: Operator plus requires its operands to have >> dimensionally equivalent units; dimensionless and first_order_rate differ >> Context: 1st expression in the 1st math element in component X >> XPath: /*[1]/*[5]/*[8]/*[1]/*[3]/*[2]/*[3] >> Unrecognised namespaces used: >> http://cellml.org/tmp-documentation >> http://imc.org/vCard/3.0# >> http://www.cellml.org/metadata/simulation/1.0# >> File is NOT valid CellML 1.0" >> >> >> The first equation it has a problem with is: >> >> d(M)/d(time) = V1 * M_ / (K1 + M_) - V2 * M / (K2 + M) >> >> where it doesn't like adding K1 to M_ (or K2 to M,) which have units >> "per minute" (first order rate) and "dimensionless" (fraction of >> concentration of:activated enzyme / total enzyme,) respectively. > > this is clearly an error in the model, and personally I'd be worried > about equations like this. For this example it is fine to give PCEnv two > stars (assuming my definition of the stars is accurate and the results > match those in the publication), but the curation status of the model is > definitely not at level 2. At best it could be level 1, but the curation > status *must* be evaluated in addition to getting the model running in > PCEnv. > >> Because of the way this model is built, getting the units to match would >> require drastically remaking it, and I don't think I'd know how to do >> it. I have some ideas, but only because this is a particularly simple >> model, with 3 differential equations and two algebraic expressions. In a >> larger model I would have absolutely no idea where to start. > > sure - I'm not suggesting you do it all....unless you really want to :-) > > what I am trying to say is that models should not be flagged as being > curated to any level unless they have actually been curated. And that > just because a model runs in one simulation tool and gives "correct" > answers does not mean it is curated. > >> So what is the solution to this? Only code up models in CellML that are >> perfect? Because technically, this model isn't valid CellML > > The solution is to only add valid curation annotations to the models in > the repository and when possible to start using Jonathan's validation > tools and JSim to provide additional checks on the model. > > At this stage, I'd be happy with giving a CellML 1.0 model without > reactions a curation status of level 2 if and only if it runs in PCEnv > and JSim without error and gives identical results (within some > numerical tolerance) and passes Jonathan's validation tests without > error. I'm suggesting PCEnv and JSim as they are the two main > cross-platform simulation tools - but any two or more would do as long > as one of them does complete units consistency checking (which I > understand is not too far away for COR). > > There is also the chance that you'll find models that are valid but fail > one or more of the tests above for some reason. These are the kinds of > models that application developers would like to see :-) > > And just a final note for the model status documentation. You seem to > add to previous documentation which in some cases starts to get a bit > misleading. For example, the jafri_rice_winslow_1998_version01_variant01 > model has probably not been validated by Penny? It would probably make > more sense to simply state the version/variant upon which the model > comes from and what the new status of the model is....if that makes sense. > This is infact a file that has been curated by Penny - she sent it to me and I uploaded it. What is confusing is the version/variant issue. Is this version 01 of variant 01, or is it variant 01 of version 03? Until this is sorted out, there appears to be no 'right' way to do it. Granted, this is in fact an inconsistency in the way i've been doing it (the latter way,) so thanks for pointing this out. > > Andre. > _______________________________________________ > cellml-discussion mailing list > [email protected] > http://www.cellml.org/mailman/listinfo/cellml-discussion Kind regards, James Lawson _______________________________________________ cellml-discussion mailing list [email protected] http://www.cellml.org/mailman/listinfo/cellml-discussion
