David More wrote:
Hi Tim,
Sorry, I read it as all the sources - not just the test script sources but I don't follow
- the CCHIT writes the test scripts - the vendor have to execute them..they are already
public or have I missed something?
No, we're using "scripts" in two different senses. In the olden days
(which still persist in a lot of places, unfortunately), software
products or installations were subject to UAT (user acceptance testing)
using scripts, in the dramatological sense, which a human
thespian/tester, sitting in front of keyboard and screen, read teh
English-language script, interpreted it, and then clicked and typed
commands to make the software do what was called for in the stage
directions in the script, checked that it seemed to do it correctly, and
then put a big tick with a pen in a checkbox next to that part of the
script....many hours or days later, testing by the human is complete.
The next day the software vendor delivers a new version, so the human
sits down and starts all over again. Boring, error-prone and expensive.
These days, the proper way to do functional testing is to use special
"test harness" software, which uses a script - in teh sense of a set of
program statements written in the test environment scripting language -
to automate the actions that that the human used to have to perform in
the bad old days. The test harness software can typically read the
screen as well, looking for particular responses or error messages.
Obviously the test script (i.e. a program) needs to be be written, in
conformance with the test specifications (which equate to the test
script for a human in the old way of doing things), but it only needs to
be written once, and thereafter only maintained or tweaked to take
account of changes in the software which is was written to test. And it
can then be re-run, again and again, at no marginal cost. Most test
harnesses provide nice summary reports when they finish running. Not
only that, but by breaking the automated test script into chunks, those
chunks can be combinatorially or randomly re-combined to do far, far
more exhaustive testing of valid actions in unusual combinations or
sequences than any human tester ever could.
For Web browser-based testing, the functional testing tool-of-choice is
probably Selenium, which runs in all the major browsers, and is open
source and free, and which has a "macro recorder" mode for Firefox to
get yo started, and can also itself be driven by other programmes. See
http://www.openqa.org/selenium/ or
http://www-128.ibm.com/developerworks/library/wa-selenium-ajax/
There are many equivalent tools for GUI applications (not all such tools
are open source, however, but none of them cost a fortune).
Obviously automation makes sense - but I suspect the costs of doing this (developing the
automation) and of all the transparency you are proposing may make the cost to the vendors
prohibitive - sounds like a lot of work to me. I suspect most of the $28,000 goes on the
testing process.
Any vendor who writes software without simultaneously writing automated
unit test and functional tests is, these days, negligent. The modern way
to write highly reliable software is to write the tests and then write
the software code. That's software engineering 101 as taught in every
university School of IT or Computer Science. So yes, there is a cost in
developing automated tests, but it is a cost software vendors and
producers should be incurring anyway, and if they are not, then their
software production methods are old-fashioned and suspect. But the other
point is that the cost of writing the automated tests is incurred once,
and after that they can be re-run again and again at no marginal cost -
unlike human-mediated tests, as CCHIT insist upon, which incur the same
high costs every time they are performed.
The fact everyone is so worried about $28K shows we don't have a very serious, broad or
deep indigenous Health IT industry - I am sure the price would not worry IBA, HCN and a
few others. In fact I believe (and this may not be popular) that we would do much better
with 4 or 5 large well resouced providers than the status quo. May be the service levels
that people are complaining about could be improved.
So you are saying that the cost of testing should be unnecessarily high
in order to drive smaller vendors out of the market? Sorry, I can't
agree with that. Nor am I sure that there is a correlation between the
size of the vendor and the quality of their software or their support
services. Perhaps an inverse relationship?
Anyway it really does not matter - if there was agreement to do something then we can work
out the details that suit all the stakeholders.
Well, no, it matters what people are agreeing to. Software accreditation
is desirable but only if done efficiently and transparently.
The bottom line is - like it or not - they have a system up and going and evolving - we
have diddly squat..that is my point - we need to get going.
Well you have my take on how to go about it. Don't look to the software
vendors or the MSIA to push this along - there is nothing in it for
them. Don't look to the Federal govt either - at least not until Mr Rudd
is in charge (no predictions as to when that will be, maybe soon, maybe
not, alas), anything that smacks of regulation will get no support in
Canberra. Sound out the Australian Consumers Association and/or some of
the learned colleges.
Tim C
On Sun, 25 Feb 2007 11:49:44 +1100, Tim Churches wrote:
David More wrote:
Hi Tim,
You and I both know no commercial providers will publish their source code - so I am
sorry your plan does not seem realistic to me.
read what I wrote - they only need to publish the source code for the
the test scripts for their software, not the source code for their
actual software. Such test scripts need reveal nothing about how their
software is actually constructed.
We can't let the public interest in teh safety of medical software be
over-ridden by software vendor preciousness about never revealing any
source code - especially when it is only test script source code that is
required, not their crown jewels. Surely you don't you want to have poor
quality non-interoperable GP software go on forever (to use your own words)?
The CCHIT is functioning as a "trusted" intermediary and as such the fact that so many
vendors have agreed to certification seems to be doing an reasonable
job.
A bloody expensive job. The results of what I propose is arguably
superior, due to much greater transparency (in teh name of public
safety), and a lot less costly for everyone involved.
I have reviewed the functionality plans and scripts and they seem pretty reasonable to
me - covering most of the bases and having an improving evolutionary
path (i.e. continuous improvment). Not perfect but way better than nothing in an
imperfect world.
They would be a good starting point for Oz accreditation standards, as
per my proposal.
The 2006 certification criteria and scripts can all be reviewed - for free - at:
http://www.cchit.org/vendors/learn/CCHIT+Ambulatory+EHR+Certification+for+2006.htm
At least they are being pretty open about the goals etc.
As they ought to be.
I see the CCHIT as a pretty cheap, well thought out, and practical approach to
improvement of the software available to the health sector and would like
something similarly practical to be done in Australia - tailored to our market
etc.
Not cheap, they ignore the possibility and desirabilty of test
automation, they are insufficiently transparent and $28k per test is not
cheap. We can do better than that.
We obviously agree on the need - I am just keen to see a path that will work
practically to get us there and Government or a surrogate to get on with it!
No need for it to be a govt body - an NGO would be acceptable provided
there was complete transparency, as per my proposal. ACA (Australian
Consumers Association, publisher of CHOICE magazine) might be an good
hosting organisation. And they are technically very competent (their
testing labs in Sydney are very impressive) and very efficient - very
little overhead or bloat. And they are increasingly harnessing consumers
via the Internet to help with testing. They may care to partner with
some medical professional bodies. A govt grant to start it off, as the
US CCHIT got, would be needed, but aim would be self-sufficiency in 3
years. Sure, there would need to be fees, but *minimal* fees due to the
use of efficient, re-usable automated testing methods, reliance on
vendors demonstrating stuff via screencasts, and leverage of the time
and effort interested third-parties via the Internet.
No need to make the accreditation mandatory for software vendors, just
make the use of accredited software mandatory for practices (after three
years to establish the process).
Tim C
On Sun, 25 Feb 2007 10:37:26 +1100, Tim Churches wrote:
David More wrote:
Hi Tim,
3 points:
1.. The fees go to allow certification to continue - not anywhere else - are
certification bodies not allowed to recover their costs?
Yes, but only the absolute minimum of costs i.e. they need to do their business in the
most efficient way, with minimal overheads. And thus no modern
offices, no hierarchy of staff, just a Web site and some email
accounts in a small back office hosted by an existing organisation. That will do. And
rely on modern, automated testing methods - see below.
Is Standards Australia meant to do it all for
free? (They a'int! and we are all being ripped off as best I can tell)
No argument there.
Frankly you need to
recognize this is the way our Government and the US seem to insist things are
organised
these days...
The recognition that Bush and Howard want to screw up the world doesn't make it right,
or that we should just roll over and acquiesce. If something is
wrong or stupid, we have a duty to say so.
my preference would have been for a totally government funded body to do all
this...but..when was the last time any government entity did this sort of
stuff for free (think TGA and its fees etc)
And NGO or QUANGO is fine for accreditation, but an efficient, lean and mean one,
which leverages modern technology and the power of the network to
achieve its ends (which are to ensure software quality, not to build its own little
empire).
2. This, very inexpensive effort in national US terms, is so far ahead of what
is
happening here (in OZ) is it grumpifying as far as I am concerned.
Yes but they spend $450 billion each year on the military in the US, and everything
else there looks cheap by comparison.
3. Note - At least one open-source solution is going for it..sorry it has to
pay but
that is the world a majority of us (under Howard and Bush) voted for -
so what can I do.? I sure didn't vote for it!
The CCHIT is happening, its working and there is 'stuff all' happening in OZ along
the
same - very important - lines. Ostrich all you like - this is
fundamentally good stuff CCHIT are doing and it is being done on the 'smell or an
oily
rag' in a relative sense.
I am told that the US govt happily pays Haliburton and other contractors $5000 each
for oily rags to be delivered to Iraq to help with the reconstruction.
Seems you want to have poor quality non-interoperable GP software to go on
forever in
OZ - or have I got it wrong and you really would like some decent
quality control etc?
No, I am absolutely in favour of formal quality assurance programmes
and/or accreditation for health-related software - more the former than the latter but
they start to merge if done correctly - iff (if and only if) the
process is both effort and financially efficient and completely transparent.
Here is how you achieve that:
a) establish a *small* unit to develop the accreditation standards in a consultative
and transparent fashion, using email and the Internet
(wikis etc), and not endless secretive meetings in capital cities with people who
don't really have much of a clue, or who have a barrow to push (or
both). Allow one year to develop Version 1.0 accreditation standards.
b) Design the accreditation standards/tests to be automatable wherever possible - and
this is most places - so the software vendors/producers can write
automated, scripted tests to demonstrate the conformance of
new versions of their code with minimal re-testing overhead. In places where
automation is not possible, then "screencast" movies, made by the vendor, of
the software performing some specified set of actions or tasks or demonstrating a
required feature should be able to be
submitted. software to record screencasts (eg Camtasia) only costs a few hundred
dollars. Any cheating by the vendor in such screencasts will be obvious,
because end users can replicate the steps shown in the screencast themselves and call
teh vendor's bluff.
c) All automated test scripts, other test code, test data, the test
results and screencasts etc must all be submitted to the accreditation body, which
runs the tests, views the screencasts, checks the
documentation and then publishes the lot on their Web site for public scrutiny. This
allows end users, public interest groups, competitors,, busy-bodies
and do-gooders to independently verify that the tests are correct and legitimate and
that no cheating has occurred. There is a
formal complaints process by which the accreditation body can be asked to investigate
evidence of cheating or anomalies or mistakes given some prima
facie evidence that such has occurred.
Given the modest size of the Australian health software market, all of the above
should only require a handful of staff to run. It leverages
the power which the Internet brings to consumer groups and end users to help the
accreditation body do its work.
Of course, software vendors may object to having their testing source
code published on the Internet, but to such an objection the answer has to be that
only test source code is required to be published - there is *no*
requirement to publish the source code of the actual software. If they object that
even such test code may reveal trade secrets, then the response has to
be that we are talking about health and medical software here, malfunctions of which
can have serious impacts on patient's lives, and thus the public
interest must override any commercial concerns over possible exposure of trade
secrets, so tough!
That's the way to do medical software accreditation.
Tim C
On Sat, 24 Feb 2007 20:32:16 +1100, Tim Churches wrote:
David More wrote:
Hi Oliver,
They are about 2 years into the program.
They are also about 1 year into certifying hospital systems.
Now that they have 40+ systems certified (at $US28,000 per time)
There has been much discussion of these fees on the international open health
list -
fees of such magnitude effectively exclude open source and community-
based
solutions. Not only that, they want the US$28k for every new version
to be re-tested. So, if a vendor puts out a minor point release, ka-ching
(sound of
cash
register), another $28k please. And their justification is that it
takes person-time to re-do the tests. Seems they've never heard of an automated test
-
write the tests once, re-run at the push of a button, which is how all
software should be tested as it is built these days. Thus, CCHIT is a farce in
practice
(Horst can supply some suitably colourful epithets here). A bit like
accreditation of general practices here in Oz, perhaps?
Tim C
On Sat, 24 Feb 2007 17:10:27 +1030, Oliver Frank wrote:
David More wrote:
Hi Oliver,
If you want to know how it can be done properly for ambulatory care (i.e. GP and
specialists) I suggest you browse www.cchit.org. They have it sorted for
the US and it is pretty impressive how they plan to move forward I reckon.
Pity GP systems is not a focus for NEHTA so this could be replicated here.
Imagine
if
there was a decent standard for functionality and interoperability
that Australian providers had to meet. They might not be all that supportive of
such
a
sensible move I fear as it might cost a few $$ and so on.
http://www.cchit.org/physicians/overview.htm
tells us:
"CCHIT is the recognized certification authority in the United States for EHR
products
-
an independent, private-sector organization that sets the Gold
Standard for EHRs."
I hope that I never hear that overworked expression 'gold standard' used again,
because
its orginal meaning is no longer known by most people.
Their PDF: "Physician's Guide: CCHIT Certification for Ambulatory Electronic
Health
Records 2006"
tells us:
"CCHIT was founded by the American Health
Information Management Association,
the Healthcare Information and Management
Systems Society and the National Alliance
for Health Information Technology.
The U.S. Department of Health and Human
Services (HHS) awarded CCHIT a three-year
contract to develop and test certification
criteria and manage an inspection process
for certifying EHRs. At the end of the
contract, CCHIT will transition to a selfsustaining
certification agency."
So they have three years of federal government money to kick start the process,
then
it
has to become self-funding. David, do you know when their three
years of government funding will be up?
"CCHIT works in collaboration with the
American Health Information Community,
the Department of Commerce's National
Institute of Standard and Technology, and
with several other organizations awarded
HHS contracts to harmonize standards,
develop prototypes for a national health
information network architecture, and assess
privacy and security laws and practices.
The work of CCHIT has been endorsed by a
number of physician professional organizations,
including:
- The American Academy of Family Physicians"
OK, so their equivalent of the RACGP is supporting it. Good.
Let's also go for three years of government funding for an organisation
indepenedent
of
government, run by the profession and software industry jointly.
Maybe we can save some time and money by using or adapting some of the standards
that
CCHIT has developed for GP computer systems in the US, keeping in mind
the very different way that medical practice is organised and funded there.
------------------------------------------------------------------------
_______________________________________________
Gpcg_talk mailing list
[email protected]
http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk
_______________________________________________
Gpcg_talk mailing list
[email protected]
http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk
__________ NOD32 2078 (20070223) Information __________
This message was checked by NOD32 antivirus system.
http://www.eset.com
__________ NOD32 2079 (20070224) Information __________
This message was checked by NOD32 antivirus system.
http://www.eset.com
__________ NOD32 2079 (20070224) Information __________
This message was checked by NOD32 antivirus system.
http://www.eset.com
_______________________________________________
Gpcg_talk mailing list
[email protected]
http://ozdocit.org/cgi-bin/mailman/listinfo/gpcg_talk
