Ok, Mark. You've set forth the philosophy for rigorous, controlled, disciplined testing. Given the health care industry, size, fragmentation, etc. what do you believe should be the approach? I've said before, and I continue to say it that there are several levels of testing and validation that need to take place - all under controlled test plans and scenarios. The first one is essential for starters since I believe there are many, many small vendors developing software for HIPAA covered entities to use that are incorporating rules for generating and interpreting the X12 rules on which the HIPAA specifications are based. So,
Test/Validation 1: Assurance that the system can correctly generate and interpret the base set of X12 rules. Without this assurance everything else is a house of cards. The system must then be able to identify and report back to the originator errors detected with sufficient detail such that the originator can identify and resolve the problem. Test/Validation 2: Assurance that the system can now generate and interpret X12 interchanges containing functional groups and transactions that both comply with the X12 rules AND the specifications of the related HIPAA implementation guides. The system must be able to identify and report back to the originator errors detected that render the interchange and included functional groups and transactions NON-COMPLIANT with the HIPAA specifications. This is also essential since there are economic penalties set forth in the law for conducting non-standard transactions. Test/Validation 3: Assurance that the system can generate and interpret X12 interchanges containing functional groups and transactions that now contain the sufficiently correct standard data that will ensure the receiving party can successfully process the transactions through its production systems to the desired outcome: payment of a claim, an accurate response to an inquiry re eligibility or claim status or referral authorization. So, then, what do you recommend the industry do to achieve these three fundamental assurances - that is, assuming you agree with them? Rachel Rachel Foerster Principal Rachel Foerster & Associates, Ltd. Professionals in EDI & Electronic Commerce 39432 North Avenue Beach Park, IL 60099 Phone: 847-872-8070 Fax: 847-872-6860 http://www.rfa-edi.com -----Original Message----- From: Mark A Lott [mailto:[EMAIL PROTECTED]] Sent: Saturday, August 10, 2002 6:28 PM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: Test Data Greetings, It seems the debate over test files gets livelier everyday, quite refreshing I think. To me it shows the difference between initial concept and actual implementation of such an initiative. I want apologize up front for the length of this e-mail but this subject is a complex one to explain in a brief synopsis. The views expressed here are my own and come from considerable time involved in the Quality Assurance world, with heavy emphasis on EDI , financial transactions and certifying millions of file formats and trading partners across the globe. I am a testing purist and feel too many organizations really don't understand the complexities and work involved in the HIPAA testing arena. Some reasons for this are from misinformation being published, trying to simplify a very time-consuming and difficult task of testing, and the belief that EDI validation is somehow the center of the testing universe even though it represents just a very small piece of the testing lifecycle. There is no such entity as an EDI Validation Test Phase, yet we choose to separate it from the overall testing process. EDI validation occurs at every level of testing and the test files really need to reflect the purpose for which it is being used in conjunction with the appropriate test phase. I for one, do not agree that free test files are needed and considering the enormous time commitment required to properly catalog and describe in detail what each file represents, it realistically cannot be accomplished in a meaningful way for the industry. This comes with one caveat, free test files would be beneficial for unit testing and file formatting in the developer arena but not really a viable alternative for a test team trying to conduct Integration, User Acceptance or Trading Partner testing. It begs the question; why do people feel test files should be free and what purpose are you trying to fulfill? Free test files will in the end cost more money than if you started from scratch. Personally, I believe that all test files should be created using production data especially for traceability and to benefit business stakeholders during the review of business functionality during the user acceptance and trading partner test phases. Creating test files using an EDI tool or via your internal translator introduces additional errors into the test verification process that will only cause expanded testing delays, timelines being missed and a misconception that your systems have been fully tested. I have also heard phrases like "my translator or software is HIPAA compliant, so we are okay". All I can do is wonder why people believe all these misconceptions. "If you tell a lie long enough, it will still never be considered a fact" Allow me to digress on this subject for people without formal QA training and expertise. A test file is derived from a test case which is derived from a set of test conditions which is created in response to testing and proving a business requirement. The test case must have an expected result in order to prove proper functionality. Free test files do not have expected results and in essence would have to be retrofitted into the test conditions. Using a free file causes test analysts to decompose the file to determine all of the pertinent contents, then adjusting the test file to meet test cases and conditions. A very important point is that test files, test cases and test conditions must have traceability to be usable, otherwise the test team has to look everywhere for defect correction when a discrepancy is found. Is the defect in the translator, the EDI validation tool, in the test file itself, in the understanding of the guides, was the business requirement written correctly, etc...? In reality, available free test files are only an infinitesimal representation of all the conditions that need to be tested to correctly ascertain whether or not business functionality has been proven. I am not sure there is great benefit in providing a test analyst 1% - 5% of their overall test file needs. There was an estimate of 1400 possible test files for Oxygen Therapy (both positive and negative test conditions), care to guess how many organizations executed even one quarter of the possible files? QA personnel should know better than to assume any particular aspect or product is 100% correct. Now, let's look at the bigger picture which will cause even more contention. If you offer free test files, through which validation engine where they executed? EDI validation tools all give different answers, but which one is right? All of them? None of them? A combination of all of them? Lets examine a real scenario: Payer X using EDI tool A to verify their files, Clearinghouse Y uses EDI tool B to verify their files, Providers X,Y and Z all use different EDI tools for their validation. All covered entities believe they are compliant because their EDI tool told them so. Provider Y sends what they believe to be a complaint file to Payer X, yet the transaction is rejected by Payer X because their front end EDI tool says it is not compliant, and so on and so forth... Can you see what the industry will uncover, but by that time it is too late. There is also discussion about having all the EDI vendors perform testing with each other, why? Shouldn't they have tested their product completely before they released it to the industry? If a company produces a quality product they should be rewarded for doing so, and quality usually comes with a price. Last I checked, this is still a free market society. This should also prove the point that EDI validation is just a small part of the information chain and cannot be seen as a so-called verification or certification point. The same goes for translators as well. These are all various pieces making up a whole and I highly doubt anyone of them are defect free. Covered Entities MUST examine testing from a business process and business functionality perspective. So many of us with years in quality assurance agree that you cannot stress enough that you must start from the known and progress to the unknown, not from unknown to unknown and NEVER assume any piece of software is 100% defect free. Therefore you should have test conditions that represent application test data, specific translator test data, EDI validation test data, etc. If you start with trying to prove business functionality in conjunction with a very robust test effort, you will uncover translator and EDI validation defects. If you start from EDI validation and go backwards, you will not find all your defects and will leave your organization exposed. How does an organization know that their systems have been fully tested? Certainly not by running free test files through their application systems, although there may very well be people doing this. By offering free files you open yourself up to many, many issues concerning viability, traceability and responsibility. Is this really the best use of volunteer time? I don't believe it is. I fear that true testing methodologies, concepts and best practices are not being promulgated enough in this industry and in this forum and organizations will have a false sense of security about their applications systems. I strongly believe that people need to really open their eyes and see what is happening in this industry and not assume everything will be taken care of for them. There are many organizations looking for help to comply with HIPAA transactions and code sets, and they look to us for assistance, we must try to not get caught up with our own business initiatives and do what it right for the industry as a whole, isn't that the definition of volunteerism, to help the greater good? This in no way reflects on the all the volunteers that have graciously given their time to work for the betterment of the industry, but a much more realistic and hard-hitting approach needs to be taken with a focus on true implementation if we are to expect this venture to be a successful one. I apologize for leaving off the sugar coating, but their needs to be a line drawn in the sand not a hole to bury one's head. This only reflects the tip of the iceberg and the frustration currently being voiced on the current testing implementation shortcomings will only grow and are very much welcome to assist in driving the point home. New education and industry initiatives are needed... If I may share a few words of wisdom: "Failure is not a single, cataclysmic event. You don't fail overnight. Instead, failure is a few errors in judgment, repeated every day." -- Jim Rohn "The quality of the solution you pick will be in direct proportion to the quantity of the solutions you consider." -- Brian Tracy Mark A Lott Please note that it may take up to 72 hours to process your request. <P>The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. Posting of advertisements or other commercial use of this listserv is specifically prohibited.
