Re: [cellml-discussion] a detailed curation specification
I have been thinking about this and I think it's worth proposing formally. But is having a whole level all about units consistency justified? Perhaps there are other things we could add to this level that could similarly require the intervention/expertise of the model author? I can't think of anything off the top of my head right now. David Nickerson wrote: Just one comment here. That is, if you look at the level 2 curation requirements, the only one that isn't satisfied by most of the models I've given two stars is the unit checking. So what this means is that we have a bunch of models which are much better curated than the level 1 curated models, but there isn't any way to actually show that if we aren't going to let them be level two. This is related to your point about splitting up the curation levels, and there are many models which would require actually reformulating the model completely to get units consistency (which would probably require the author getting involved.) If we did move units consistency up to level three, I think it would make things more straight forward. Rather than moving units consistency up to level 3 it would probably be better to move what is currently level 3 up to 4 and make level 3 all about units consistency. I think for the time being I'm going to take a left-wing approach and spend more time fixing the models that are completely broken. which I think is the right thing to be doing, I just think we need to be careful that the status advertised for a given model matches the definition of that status. Yep. I feel like biting off something I can chew right now. David. ___ cellml-discussion mailing list cellml-discussion@cellml.org http://www.cellml.org/mailman/listinfo/cellml-discussion
Re: [cellml-discussion] a detailed curation specification
I have been thinking about this and I think it's worth proposing formally. But is having a whole level all about units consistency justified? Perhaps there are other things we could add to this level that could similarly require the intervention/expertise of the model author? I can't think of anything off the top of my head right now. Personally I think there is nothing wrong with the current levels and keeping units with getting the model giving correct results. I have little faith in a model which can give the correct results while being defined with inconsistent units. I was simply suggesting moving units to a higher level to try and ease the burden of getting models beyond level 1 curation, as you pointed out a model which does run and gives reasonable results is much better than one which does not run at all, even with inconsistent units, depending upon the task you wish to put it to. The units inconsistency issue is a legacy of the majority of models being written by hand with no way to test them. In most cases the models already in the repository can now be tested with either JSim or PyCML for units consistency, so it would be good to see those tests being done as part of your curation workflow - even if that just ends up as a comment in the model status that the units are not consistent or something. David. ___ cellml-discussion mailing list cellml-discussion@cellml.org http://www.cellml.org/mailman/listinfo/cellml-discussion
Re: [cellml-discussion] a detailed curation specification
David Nickerson wrote: I have been thinking about this and I think it's worth proposing formally. But is having a whole level all about units consistency justified? Perhaps there are other things we could add to this level that could similarly require the intervention/expertise of the model author? I can't think of anything off the top of my head right now. Personally I think there is nothing wrong with the current levels and keeping units with getting the model giving correct results. I have little faith in a model which can give the correct results while being defined with inconsistent units. I was simply suggesting moving units to a higher level to try and ease the burden of getting models beyond level 1 curation, as you pointed out a model which does run and gives reasonable results is much better than one which does not run at all, even with inconsistent units, depending upon the task you wish to put it to. The units inconsistency issue is a legacy of the majority of models being written by hand with no way to test them. In most cases the models already in the repository can now be tested with either JSim or PyCML for units consistency, so it would be good to see those tests being done as part of your curation workflow - even if that just ends up as a comment in the model status that the units are not consistent or something. Sure, I'll start doing that. By the way, what exactly is a workflow? ;) David. ___ cellml-discussion mailing list cellml-discussion@cellml.org http://www.cellml.org/mailman/listinfo/cellml-discussion ___ cellml-discussion mailing list cellml-discussion@cellml.org http://www.cellml.org/mailman/listinfo/cellml-discussion
Re: [cellml-discussion] a detailed curation specification
Just one comment here. That is, if you look at the level 2 curation requirements, the only one that isn't satisfied by most of the models I've given two stars is the unit checking. So what this means is that we have a bunch of models which are much better curated than the level 1 curated models, but there isn't any way to actually show that if we aren't going to let them be level two. This is related to your point about splitting up the curation levels, and there are many models which would require actually reformulating the model completely to get units consistency (which would probably require the author getting involved.) If we did move units consistency up to level three, I think it would make things more straight forward. I think for the time being I'm going to take a left-wing approach and spend more time fixing the models that are completely broken. David Nickerson wrote: http://www.cellml.org/repository-info/info essentially lays out the requirements for each level of curation. I think this gives a clear indication of what is required for level 0, 1, and 2 curation and my document you mention below starts to look at what is required for level 3. Just to point out that that document is the combination of discussions with many people, primarily based on a proposal from Jim Bassingthwaighte. I think until there is some progress on defining what level 3 curation is that we don't worry about it for now, especially since it is generally regarded that no existing published models would meet all the requirements for level 3 curation. I think level 3 is something we can aspire to as people start using tools like CellML and decent testing frameworks in the actual development of new models rather than an add-on after the model is already developed (like what Randy was talking about at the CellML workshop). In terms of level 1 and 2 curation, I'm not really sure what more detail you need in the way of a specification? Currently the only way to make sure the math described by a CellML model is equivalent to a published paper is to sit down and look at them both side by side. Whether you do that looking at the content MathML directly or use some utility to render the MathML in a more readable format is a personal preference. Ideally this is best done by the model author, but any model curator can undertake this level of curation. Similarly, how level 2 curation is achieved is really up to the person doing the curation. I've put forward a suggestion on how this could be done, but different people can choose to use whatever tools they prefer as long as there is confidence that the required conditions have been met. Personally, I'd say the more different simulation and validation tools used the better, although the formulation and mathematics used in a given model might restrict which tools can be used and force more manual checking upon the curator. Ideally I think level 2 curation should be undertaken independently of the model author, but can't really see a reason to enforce this as the curation annotation should contain the curator data. In fact, there is no reason curation can't be done by more than one curator, whereby model users might have more confidence in a model that has been stamped level 2 by more than one person... A key point to note is that a model curated to level 2 is not necessarily also curated to level 1 - so we need to be careful about the graphical representation used on in the model repository. i.e., the first two stars being yellow should imply that the model satisfies bother level 1 and 2 curation. In general, at least for the electrophysiology models, none of the models in the repository will satisfy both curation levels 1 and 2. Also, the requirements for each of the curation levels are currently the result of many discussions on what we'd ideally like to see for each level. As people spend more time curating models and using these annotations I'd expect there could be some changes in these definitions required. So if you have any suggestions please put them forward. One that has come up a bit would be breaking level 2 into more pieces, for example pushing the units consistency requirement to a higher level. David. PS - no immediate plans to come back to Auckland, I think the 8th-18th comment was in regard to the April workshop. James Lawson wrote: Dear all, Another thing Peter has asked me to seek input on is the need for a detailed, realistic curation specification that I can follow. Andre's 'thoughts on model curation and model repositories' goes some way towards this but it isn't formalised and this kind of document probably needs the input of more than one person. The proposed document also needs to take into account the current and future state of the repository, and provide guidelines to follow that will result in significant, consistent progress
