Mario Ruiz wrote: > Hi Tim, > > In the use of the term "vendor", are you referring to all software > providers?. Including commercial, non-commercial and open source > software providers which supply and make available software to any > medical practice whether the software is paid for or not.
Yes. > If that is the case, then, would you preclude from the certification any > software that, a person/doctor/afficionado may write or modify for a > practice?. Would that make the whole practice non-conforming as > mentioned in the thread? No, such modified versions could also be submitted for accreditation, with modified versions of the test scripts if necessary. However, for open source software, it may be possible to allow "delta" accreditation, in which the degree and nature of changes to an already accredited version are considered and only those aspects affected by the changes need be re-accredited. Would need some thought. > Who would do and pay for certification of in-house developed or free > software that works just fine? Whoever wants it to be accredited will have to pay for its accreditation. Of course, the rhetorical question must be asked: without comprehensive, automated tests, how does one know that the software works properly? Is "seems to work" (as in "I tried it once or twice and it didn't crash using the particular test data I gave it, and the answers or stored data looked more or less sensible although I didn't bother to exhaustively verify every data item") good enough? I don't think so. One of the motivations behind my proposal is not just to require conformance with what is ultimately an arbitrary checklist (the accreditation specifications), but also to encourage software vendors/producers to adopt thorough, modern automated software testing procedures which go beyond just gaining accreditation. > Do you propose exceptions to the rule? No. But note that software accreditation is not compulsory - I would propose that one can still sell and/or use non-accredited medical software, but practices using non-accredited software miss out on the points allocated for "uses accredited software" if/when their practice is accredited. Tim C > Tim Churches wrote: >> David More wrote: >>> Hi Tim, >>> >>> Sorry, I read it as all the sources - not just the test script >>> sources but I don't follow - the CCHIT writes the test scripts - the >>> vendor have to execute them..they are already public or have I missed >>> something? >> >> No, we're using "scripts" in two different senses. In the olden days >> (which still persist in a lot of places, unfortunately), software >> products or installations were subject to UAT (user acceptance testing) >> using scripts, in the dramatological sense, which a human >> thespian/tester, sitting in front of keyboard and screen, read teh >> English-language script, interpreted it, and then clicked and typed >> commands to make the software do what was called for in the stage >> directions in the script, checked that it seemed to do it correctly, and >> then put a big tick with a pen in a checkbox next to that part of the >> script....many hours or days later, testing by the human is complete. >> The next day the software vendor delivers a new version, so the human >> sits down and starts all over again. Boring, error-prone and expensive. >> >> These days, the proper way to do functional testing is to use special >> "test harness" software, which uses a script - in teh sense of a set of >> program statements written in the test environment scripting language - >> to automate the actions that that the human used to have to perform in >> the bad old days. The test harness software can typically read the >> screen as well, looking for particular responses or error messages. >> Obviously the test script (i.e. a program) needs to be be written, in >> conformance with the test specifications (which equate to the test >> script for a human in the old way of doing things), but it only needs to >> be written once, and thereafter only maintained or tweaked to take >> account of changes in the software which is was written to test. And it >> can then be re-run, again and again, at no marginal cost. Most test >> harnesses provide nice summary reports when they finish running. Not >> only that, but by breaking the automated test script into chunks, those >> chunks can be combinatorially or randomly re-combined to do far, far >> more exhaustive testing of valid actions in unusual combinations or >> sequences than any human tester ever could. >> >> For Web browser-based testing, the functional testing tool-of-choice is >> probably Selenium, which runs in all the major browsers, and is open >> source and free, and which has a "macro recorder" mode for Firefox to >> get yo started, and can also itself be driven by other programmes. See >> http://www.openqa.org/selenium/ or >> http://www-128.ibm.com/developerworks/library/wa-selenium-ajax/ >> >> There are many equivalent tools for GUI applications (not all such tools >> are open source, however, but none of them cost a fortune). >> >>> Obviously automation makes sense - but I suspect the costs of doing >>> this (developing the automation) and of all the transparency you are >>> proposing may make the cost to the vendors prohibitive - sounds like >>> a lot of work to me. I suspect most of the $28,000 goes on the >>> testing process. >> >> Any vendor who writes software without simultaneously writing automated >> unit test and functional tests is, these days, negligent. The modern way >> to write highly reliable software is to write the tests and then write >> the software code. That's software engineering 101 as taught in every >> university School of IT or Computer Science. So yes, there is a cost in >> developing automated tests, but it is a cost software vendors and >> producers should be incurring anyway, and if they are not, then their >> software production methods are old-fashioned and suspect. But the other >> point is that the cost of writing the automated tests is incurred once, >> and after that they can be re-run again and again at no marginal cost - >> unlike human-mediated tests, as CCHIT insist upon, which incur the same >> high costs every time they are performed. >> >>> The fact everyone is so worried about $28K shows we don't have a very >>> serious, broad or deep indigenous Health IT industry - I am sure the >>> price would not worry IBA, HCN and a few others. In fact I believe >>> (and this may not be popular) that we would do much better with 4 or >>> 5 large well resouced providers than the status quo. May be the >>> service levels that people are complaining about could be improved. >> >> So you are saying that the cost of testing should be unnecessarily high >> in order to drive smaller vendors out of the market? Sorry, I can't >> agree with that. Nor am I sure that there is a correlation between the >> size of the vendor and the quality of their software or their support >> services. Perhaps an inverse relationship? >> >>> Anyway it really does not matter - if there was agreement to do >>> something then we can work out the details that suit all the >>> stakeholders. >> >> Well, no, it matters what people are agreeing to. Software accreditation >> is desirable but only if done efficiently and transparently. >> >>> The bottom line is - like it or not - they have a system up and going >>> and evolving - we have diddly squat..that is my point - we need to >>> get going. >> >> Well you have my take on how to go about it. Don't look to the software >> vendors or the MSIA to push this along - there is nothing in it for >> them. Don't look to the Federal govt either - at least not until Mr Rudd >> is in charge (no predictions as to when that will be, maybe soon, maybe >> not, alas), anything that smacks of regulation will get no support in >> Canberra. Sound out the Australian Consumers Association and/or some of >> the learned colleges. >> >> Tim C >> >>> On Sun, 25 Feb 2007 11:49:44 +1100, Tim Churches wrote: >>>> David More wrote: >>>>> Hi Tim, >>>>> >>>>> You and I both know no commercial providers will publish their >>>>> source code - so I am >>> sorry your plan does not seem realistic to me. >>>> read what I wrote - they only need to publish the source code for the >>>> the test scripts for their software, not the source code for their >>>> actual software. Such test scripts need reveal nothing about how their >>>> software is actually constructed. >>>> >>>> We can't let the public interest in teh safety of medical software be >>>> over-ridden by software vendor preciousness about never revealing any >>>> source code - especially when it is only test script source code >>>> that is >>>> required, not their crown jewels. Surely you don't you want to have >>>> poor >>>> quality non-interoperable GP software go on forever (to use your own >>>> words)? >>>> >>>>> The CCHIT is functioning as a "trusted" intermediary and as such >>>>> the fact that so many >>> vendors have agreed to certification seems to be doing an reasonable >>>>> job. >>>>> >>>> A bloody expensive job. The results of what I propose is arguably >>>> superior, due to much greater transparency (in teh name of public >>>> safety), and a lot less costly for everyone involved. >>>> >>>>> I have reviewed the functionality plans and scripts and they seem >>>>> pretty reasonable to >>> me - covering most of the bases and having an improving evolutionary >>>>> path (i.e. continuous improvment). Not perfect but way better than >>>>> nothing in an >>> imperfect world. >>>> They would be a good starting point for Oz accreditation standards, as >>>> per my proposal. >>>> >>>>> The 2006 certification criteria and scripts can all be reviewed - >>>>> for free - at: >>>>> >>>>> http://www.cchit.org/vendors/learn/CCHIT+Ambulatory+EHR+Certification+for+2006.htm >>>>> >>>>> >>>>> At least they are being pretty open about the goals etc. >>>>> >>>> As they ought to be. >>>> >>>>> I see the CCHIT as a pretty cheap, well thought out, and practical >>>>> approach to >>> improvement of the software available to the health sector and would >>> like >>>>> something similarly practical to be done in Australia - tailored to >>>>> our market etc. >>>>> >>>> Not cheap, they ignore the possibility and desirabilty of test >>>> automation, they are insufficiently transparent and $28k per test is >>>> not >>>> cheap. We can do better than that. >>>> >>>>> We obviously agree on the need - I am just keen to see a path that >>>>> will work >>> practically to get us there and Government or a surrogate to get on >>> with it! >>>> No need for it to be a govt body - an NGO would be acceptable provided >>>> there was complete transparency, as per my proposal. ACA (Australian >>>> Consumers Association, publisher of CHOICE magazine) might be an good >>>> hosting organisation. And they are technically very competent (their >>>> testing labs in Sydney are very impressive) and very efficient - very >>>> little overhead or bloat. And they are increasingly harnessing >>>> consumers >>>> via the Internet to help with testing. They may care to partner with >>>> some medical professional bodies. A govt grant to start it off, as the >>>> US CCHIT got, would be needed, but aim would be self-sufficiency in 3 >>>> years. Sure, there would need to be fees, but *minimal* fees due to the >>>> use of efficient, re-usable automated testing methods, reliance on >>>> vendors demonstrating stuff via screencasts, and leverage of the time >>>> and effort interested third-parties via the Internet. >>>> >>>> No need to make the accreditation mandatory for software vendors, just >>>> make the use of accredited software mandatory for practices (after >>>> three >>>> years to establish the process). >>>> >>>> Tim C >>>> >>>>> On Sun, 25 Feb 2007 10:37:26 +1100, Tim Churches wrote: >>>>>> David More wrote: >>>>>>> Hi Tim, >>>>>>> >>>>>>> 3 points: >>>>>>> >>>>>>> 1.. The fees go to allow certification to continue - not anywhere >>>>>>> else - are >>>>>>> >>>>> certification bodies not allowed to recover their costs? >>>>>> Yes, but only the absolute minimum of costs i.e. they need to do >>>>>> their business in the >>> most efficient way, with minimal overheads. And thus no modern >>>>>> offices, no hierarchy of staff, just a Web site and some email >>>>>> accounts in a small back office hosted by an existing >>>>>> organisation. That will do. And >>> rely on modern, automated testing methods - see below. >>>>>>> Is Standards Australia meant to do it all for >>>>>>> free? (They a'int! and we are all being ripped off as best I can >>>>>>> tell) >>>>>>> >>>>>> No argument there. >>>>>> >>>>>>> Frankly you need to >>>>>>> recognize this is the way our Government and the US seem to >>>>>>> insist things are >>> organised >>>>> these days... >>>>>> The recognition that Bush and Howard want to screw up the world >>>>>> doesn't make it right, >>> or that we should just roll over and acquiesce. If something is >>>>>> wrong or stupid, we have a duty to say so. >>>>>> >>>>>>> my preference would have been for a totally government funded >>>>>>> body to do all >>>>>>> >>>>> this...but..when was the last time any government entity did this >>>>> sort of >>>>> >>>>>>> stuff for free (think TGA and its fees etc) >>>>>>> >>>>>> And NGO or QUANGO is fine for accreditation, but an efficient, >>>>>> lean and mean one, >>> which leverages modern technology and the power of the network to >>>>>> achieve its ends (which are to ensure software quality, not to >>>>>> build its own little >>> empire). >>>>>>> 2. This, very inexpensive effort in national US terms, is so far >>>>>>> ahead of what is >>>>>>> >>>>> happening here (in OZ) is it grumpifying as far as I am concerned. >>>>>> Yes but they spend $450 billion each year on the military in the >>>>>> US, and everything >>> else there looks cheap by comparison. >>>>>>> 3. Note - At least one open-source solution is going for >>>>>>> it..sorry it has to pay but >>>>>>> >>>>> that is the world a majority of us (under Howard and Bush) voted >>>>> for - >>>>> >>>>>>> so what can I do.? I sure didn't vote for it! >>>>>>> >>>>>>> The CCHIT is happening, its working and there is 'stuff all' >>>>>>> happening in OZ along >>> the >>>>> same - very important - lines. Ostrich all you like - this is >>>>>>> fundamentally good stuff CCHIT are doing and it is being done on >>>>>>> the 'smell or an >>> oily >>>>> rag' in a relative sense. >>>>>> I am told that the US govt happily pays Haliburton and other >>>>>> contractors $5000 each >>> for oily rags to be delivered to Iraq to help with the reconstruction. >>>>>>> Seems you want to have poor quality non-interoperable GP software >>>>>>> to go on forever in >>>>>>> >>>>> OZ - or have I got it wrong and you really would like some decent >>>>>>> quality control etc? >>>>>>> >>>>>> No, I am absolutely in favour of formal quality assurance programmes >>>>>> and/or accreditation for health-related software - more the former >>>>>> than the latter but >>> they start to merge if done correctly - iff (if and only if) the >>>>>> process is both effort and financially efficient and completely >>>>>> transparent. >>>>>> >>>>>> Here is how you achieve that: >>>>>> >>>>>> a) establish a *small* unit to develop the accreditation standards >>>>>> in a consultative >>> and transparent fashion, using email and the Internet >>>>>> (wikis etc), and not endless secretive meetings in capital cities >>>>>> with people who >>> don't really have much of a clue, or who have a barrow to push (or >>>>>> both). Allow one year to develop Version 1.0 accreditation standards. >>>>>> >>>>>> b) Design the accreditation standards/tests to be automatable >>>>>> wherever possible - and >>> this is most places - so the software vendors/producers can write >>>>>> automated, scripted tests to demonstrate the conformance of >>>>>> new versions of their code with minimal re-testing overhead. In >>>>>> places where >>> automation is not possible, then "screencast" movies, made by the >>> vendor, of >>>>>> the software performing some specified set of actions or tasks or >>>>>> demonstrating a >>> required feature should be able to be >>>>>> submitted. software to record screencasts (eg Camtasia) only costs >>>>>> a few hundred >>> dollars. Any cheating by the vendor in such screencasts will be obvious, >>>>>> because end users can replicate the steps shown in the screencast >>>>>> themselves and call >>> teh vendor's bluff. >>>>>> c) All automated test scripts, other test code, test data, the test >>>>>> results and screencasts etc must all be submitted to the >>>>>> accreditation body, which >>> runs the tests, views the screencasts, checks the >>>>>> documentation and then publishes the lot on their Web site for >>>>>> public scrutiny. This >>> allows end users, public interest groups, competitors,, busy-bodies >>>>>> and do-gooders to independently verify that the tests are correct >>>>>> and legitimate and >>> that no cheating has occurred. There is a >>>>>> formal complaints process by which the accreditation body can be >>>>>> asked to investigate >>> evidence of cheating or anomalies or mistakes given some prima >>>>>> facie evidence that such has occurred. >>>>>> >>>>>> Given the modest size of the Australian health software market, >>>>>> all of the above >>> should only require a handful of staff to run. It leverages >>>>>> the power which the Internet brings to consumer groups and end >>>>>> users to help the >>> accreditation body do its work. >>>>>> Of course, software vendors may object to having their testing source >>>>>> code published on the Internet, but to such an objection the >>>>>> answer has to be that >>> only test source code is required to be published - there is *no* >>>>>> requirement to publish the source code of the actual software. If >>>>>> they object that >>> even such test code may reveal trade secrets, then the response has to >>>>>> be that we are talking about health and medical software here, >>>>>> malfunctions of which >>> can have serious impacts on patient's lives, and thus the public >>>>>> interest must override any commercial concerns over possible >>>>>> exposure of trade >>> secrets, so tough! >>>>>> That's the way to do medical software accreditation. >>>>>> >>>>>> Tim C >>>>>> >>>>>>> On Sat, 24 Feb 2007 20:32:16 +1100, Tim Churches wrote: >>>>>>>> David More wrote: >>>>>>>>> Hi Oliver, >>>>>>>>> >>>>>>>>> They are about 2 years into the program. >>>>>>>>> >>>>>>>>> They are also about 1 year into certifying hospital systems. >>>>>>>>> >>>>>>>>> Now that they have 40+ systems certified (at $US28,000 per time) >>>>>>>>> >>>>>>>> There has been much discussion of these fees on the >>>>>>>> international open health list - >>>>>>>> >>>>> fees of such magnitude effectively exclude open source and community- >>>>>>>> based >>>>>>>> >>>>>>> solutions. Not only that, they want the US$28k for every new version >>>>>>>> to be re-tested. So, if a vendor puts out a minor point release, >>>>>>>> ka-ching (sound of >>>>>>>> >>>>> cash >>>>>>> register), another $28k please. And their justification is that it >>>>>>>> takes person-time to re-do the tests. Seems they've never heard >>>>>>>> of an automated test >>> - >>>>>>> write the tests once, re-run at the push of a button, which is >>>>>>> how all >>>>>>> >>>>>>>> software should be tested as it is built these days. Thus, CCHIT >>>>>>>> is a farce in >>>>>>>> >>>>> practice >>>>>>> (Horst can supply some suitably colourful epithets here). A bit like >>>>>>>> accreditation of general practices here in Oz, perhaps? >>>>>>>> >>>>>>>> Tim C >>>>>>>> >>>>>>>>> On Sat, 24 Feb 2007 17:10:27 +1030, Oliver Frank wrote: >>>>>>>>>> David More wrote: >>>>>>>>>>> Hi Oliver, >>>>>>>>>>> >>>>>>>>>>> If you want to know how it can be done properly for >>>>>>>>>>> ambulatory care (i.e. GP and >>>>>>>>>>> >>>>>>>>> specialists) I suggest you browse www.cchit.org. They have it >>>>>>>>> sorted for >>>>>>>>> >>>>>>>>>>> the US and it is pretty impressive how they plan to move >>>>>>>>>>> forward I reckon. >>>>>>>>>>> >>>>>>>>>>> Pity GP systems is not a focus for NEHTA so this could be >>>>>>>>>>> replicated here. >>> Imagine >>>>> if >>>>>>>>> there was a decent standard for functionality and interoperability >>>>>>>>>>> that Australian providers had to meet. They might not be all >>>>>>>>>>> that supportive of >>>>>>>>>>> >>>>> such >>>>>>> a >>>>>>>>> sensible move I fear as it might cost a few $$ and so on. >>>>>>>>>> http://www.cchit.org/physicians/overview.htm >>>>>>>>>> >>>>>>>>>> tells us: >>>>>>>>>> >>>>>>>>>> "CCHIT is the recognized certification authority in the United >>>>>>>>>> States for EHR >>>>>>>>>> >>>>> products >>>>>>> - >>>>>>>>> an independent, private-sector organization that sets the Gold >>>>>>>>>> Standard for EHRs." >>>>>>>>>> >>>>>>>>>> I hope that I never hear that overworked expression 'gold >>>>>>>>>> standard' used again, >>>>>>>>>> >>>>>>> because >>>>>>>>> its orginal meaning is no longer known by most people. >>>>>>>>>> Their PDF: "Physician's Guide: CCHIT Certification for >>>>>>>>>> Ambulatory Electronic >>> Health >>>>>>>>> Records 2006" >>>>>>>>>> tells us: >>>>>>>>>> >>>>>>>>>> "CCHIT was founded by the American Health >>>>>>>>>> Information Management Association, >>>>>>>>>> the Healthcare Information and Management >>>>>>>>>> Systems Society and the National Alliance >>>>>>>>>> for Health Information Technology. >>>>>>>>>> The U.S. Department of Health and Human >>>>>>>>>> Services (HHS) awarded CCHIT a three-year >>>>>>>>>> contract to develop and test certification >>>>>>>>>> criteria and manage an inspection process >>>>>>>>>> for certifying EHRs. At the end of the >>>>>>>>>> contract, CCHIT will transition to a selfsustaining >>>>>>>>>> certification agency." >>>>>>>>>> >>>>>>>>>> So they have three years of federal government money to kick >>>>>>>>>> start the process, >>> then >>>>>>> it >>>>>>>>> has to become self-funding. David, do you know when their three >>>>>>>>>> years of government funding will be up? >>>>>>>>>> >>>>>>>>>> "CCHIT works in collaboration with the >>>>>>>>>> American Health Information Community, >>>>>>>>>> the Department of Commerce's National >>>>>>>>>> Institute of Standard and Technology, and >>>>>>>>>> with several other organizations awarded >>>>>>>>>> HHS contracts to harmonize standards, >>>>>>>>>> develop prototypes for a national health >>>>>>>>>> information network architecture, and assess >>>>>>>>>> privacy and security laws and practices. >>>>>>>>>> The work of CCHIT has been endorsed by a >>>>>>>>>> number of physician professional organizations, >>>>>>>>>> including: >>>>>>>>>> - The American Academy of Family Physicians" >>>>>>>>>> >>>>>>>>>> OK, so their equivalent of the RACGP is supporting it. Good. >>>>>>>>>> >>>>>>>>>> Let's also go for three years of government funding for an >>>>>>>>>> organisation >>> indepenedent >>>>>>> of >>>>>>>>> government, run by the profession and software industry jointly. >>>>>>>>>> Maybe we can save some time and money by using or adapting >>>>>>>>>> some of the standards >>>>>>>>>> >>>>> that >>>>>>>>> CCHIT has developed for GP computer systems in the US, keeping >>>>>>>>> in mind >>>>>>>>> >>>>>>>>>> the very different way that medical practice is organised and >>>>>>>>>> funded there. >>>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gpcg_talk mailing list >>>>>>>>> [email protected] >>>>>>>>> http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gpcg_talk mailing list >>>>>>>> [email protected] >>>>>>>> http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk >>>>>>>> >>>>>>>> __________ NOD32 2078 (20070223) Information __________ >>>>>>>> >>>>>>>> This message was checked by NOD32 antivirus system. >>>>>>>> http://www.eset.com >>>>>>>> >>>>>> __________ NOD32 2079 (20070224) Information __________ >>>>>> >>>>>> This message was checked by NOD32 antivirus system. >>>>>> http://www.eset.com >>>> __________ NOD32 2079 (20070224) Information __________ >>>> >>>> This message was checked by NOD32 antivirus system. >>>> http://www.eset.com >> >> _______________________________________________ >> Gpcg_talk mailing list >> [email protected] >> http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk >> >> >> > _______________________________________________ Gpcg_talk mailing list [email protected] http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk
