Rachel, Test/Validation 1: Assurance that the system can correctly generate and interpret the base set of X12 rules. Without this assurance everything else is a house of cards. The system must then be able to identify and report back to the originator errors detected with sufficient detail such that the originator can identify and resolve the problem.
#1 - Yes I agree with this test scenario although its implementation is not cut and dry. Several issues arise here, the validation software being used today gives different results, how will you ever know you are completely correct? Secondly, there is a large test data need in order to fully and robustly test all the various conditions. If you are creating software then you must test it for all possible variations and implementations, hence the need for a structured decomposition of the software and subsequent test case and test file creation process. Test/Validation 2: Assurance that the system can now generate and interpret X12 interchanges containing functional groups and transactions that both comply with the X12 rules AND the specifications of the related HIPAA implementation guides. The system must be able to identify and report back to the originator errors detected that render the interchange and included functional groups and transactions NON-COMPLIANT with the HIPAA specifications. This is also essential since there are economic penalties set forth in the law for conducting non-standard transactions. #2 - All the above issues and the creation of structured negative test conditions specifically designed to cause front-end rejection by the validation software and the translators. Test/Validation 3: Assurance that the system can generate and interpret X12 interchanges containing functional groups and transactions that now contain the sufficiently correct standard data that will ensure the receiving party can successfully process the transactions through its production systems to the desired outcome: payment of a claim, an accurate response to an inquiry re eligibility or claim status or referral authorization. #3 - This requires massive amounts of production test data to prove business process compliance. Stakeholders will be reviewing internal test results to determine compliance and whether or not their applications are functioning properly. Of the three examples, the only one that has a baseline is the production system. Even though a production system may contain defects, they are known entities and can be accounted for in the expected results and concurred by the business. Validation and translators have unknown errors and in their current state some of them cannot be used as a baseline. So, then, what do you recommend the industry do to achieve these three fundamental assurances - that is, assuming you agree with them? Options- 1) Prior to contracting with a validation service maybe clients should review internal reports on the testing effort of validation software and see if the defects discovered have been corrected and review the robustness of the test effort in general. If validation companies are preaching testing methodologies and come across as testing experts then shouldn't they have taken what they preach and used it internally first? Wouldn't you think they would have had to use an extremely large test file repository to test their own software, since it must accurately report on every possible defect in X12 and the IG's? The answers may surprise you..... I personally use validators to provide feedback for remediation but under no condition will I treat what they say as gospel. Validation software cannot certify anything until the software that runs it has been certified as functioning properly. 2) In regards to translators, the above statement should be true but to a lesser extent, since client mapping is involved and they cannot truly be completely tested on behalf of the covered entity until the mapping has been completed. Although, a translator can be tested for an off-the-shelf compliance for X12 and IG's. What an end user does with the product after it has been installed is a whole different story. 3) For software vendors, they can also create an off-the-shelf compliance by thoroughly testing their products prior to release. If the software is just installed without modifications then the compliance still holds, if changes are made then testing from a covered entity perspective must be performed on-site. 4) A topic not talked about enough is the level of testing that clearinghouses have to perform, maybe there should be a whitepaper on that topic alone. 5) Since I believe in a free market system, I don't believe WEDI should do anything on behalf of free test files. Who volunteered WEDI to do that in the first place? Myself and other companies happen to sell test files and we strictly prohibit their use outside of an entity. Why should WEDI give something for free? They don't give free translators or free EDI validation products for public consumption, the same should hold true for any marketable product or service. If individual companies want to go through all the effort to catalog, document and give away free files then let them do it on their own and deal with any circumstances arising from their decision, but do not expose WEDI to any adverse situations that may arise. 6) HIPAA is unique in one regard, organizations cannot just release software with defects and fix the rest in production like they do with most any other release. If these entities are using old paradigms for testing, then it is time for them to realize that HIPAA does require an extremely structured and robust testing effort. Concepts like test automation and test management tools should be considered since HIPAA remediation will continue for quite some time and they will need reusability for all their testing efforts. I don't foresee the phrase, "we are 80% compliant or 90% compliant", and you either are or are not. 7) I also think there should much more involvement in this forum from quality assurance and testing professionals, resources that have tested large legacy and client server applications and know the amount of work involved to prove a product is in compliance or certifiable. I have made a concerted push to my fellow QA professionals for greater involvement and gratefully I can say that they will become more prominent within the next few weeks. 8) I don't feel I am in the minority and by the number of people I have discussed this issue with, they all agree something must be done, done quickly and done right. The solutions to these issues lie within the industry itself and its ability to self regulate, there are some great ideas being circulated and I hope to see them realized very shortly. Mark A Lott -----Original Message----- From: Rachel Foerster [mailto:[EMAIL PROTECTED]] Sent: Sunday, August 11, 2002 9:55 AM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: Test Data Ok, Mark. You've set forth the philosophy for rigorous, controlled, disciplined testing. Given the health care industry, size, fragmentation, etc. what do you believe should be the approach? I've said before, and I continue to say it that there are several levels of testing and validation that need to take place - all under controlled test plans and scenarios. The first one is essential for starters since I believe there are many, many small vendors developing software for HIPAA covered entities to use that are incorporating rules for generating and interpreting the X12 rules on which the HIPAA specifications are based. So, Test/Validation 1: Assurance that the system can correctly generate and interpret the base set of X12 rules. Without this assurance everything else is a house of cards. The system must then be able to identify and report back to the originator errors detected with sufficient detail such that the originator can identify and resolve the problem. Test/Validation 2: Assurance that the system can now generate and interpret X12 interchanges containing functional groups and transactions that both comply with the X12 rules AND the specifications of the related HIPAA implementation guides. The system must be able to identify and report back to the originator errors detected that render the interchange and included functional groups and transactions NON-COMPLIANT with the HIPAA specifications. This is also essential since there are economic penalties set forth in the law for conducting non-standard transactions. Test/Validation 3: Assurance that the system can generate and interpret X12 interchanges containing functional groups and transactions that now contain the sufficiently correct standard data that will ensure the receiving party can successfully process the transactions through its production systems to the desired outcome: payment of a claim, an accurate response to an inquiry re eligibility or claim status or referral authorization. So, then, what do you recommend the industry do to achieve these three fundamental assurances - that is, assuming you agree with them? Rachel Rachel Foerster Principal Rachel Foerster & Associates, Ltd. Professionals in EDI & Electronic Commerce 39432 North Avenue Beach Park, IL 60099 Phone: 847-872-8070 Fax: 847-872-6860 http://www.rfa-edi.com -----Original Message----- From: Mark A Lott [mailto:[EMAIL PROTECTED]] Sent: Saturday, August 10, 2002 6:28 PM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: Test Data Greetings, It seems the debate over test files gets livelier everyday, quite refreshing I think. To me it shows the difference between initial concept and actual implementation of such an initiative. I want apologize up front for the length of this e-mail but this subject is a complex one to explain in a brief synopsis. The views expressed here are my own and come from considerable time involved in the Quality Assurance world, with heavy emphasis on EDI , financial transactions and certifying millions of file formats and trading partners across the globe. I am a testing purist and feel too many organizations really don't understand the complexities and work involved in the HIPAA testing arena. Some reasons for this are from misinformation being published, trying to simplify a very time-consuming and difficult task of testing, and the belief that EDI validation is somehow the center of the testing universe even though it represents just a very small piece of the testing lifecycle. There is no such entity as an EDI Validation Test Phase, yet we choose to separate it from the overall testing process. EDI validation occurs at every level of testing and the test files really need to reflect the purpose for which it is being used in conjunction with the appropriate test phase. I for one, do not agree that free test files are needed and considering the enormous time commitment required to properly catalog and describe in detail what each file represents, it realistically cannot be accomplished in a meaningful way for the industry. This comes with one caveat, free test files would be beneficial for unit testing and file formatting in the developer arena but not really a viable alternative for a test team trying to conduct Integration, User Acceptance or Trading Partner testing. It begs the question; why do people feel test files should be free and what purpose are you trying to fulfill? Free test files will in the end cost more money than if you started from scratch. Personally, I believe that all test files should be created using production data especially for traceability and to benefit business stakeholders during the review of business functionality during the user acceptance and trading partner test phases. Creating test files using an EDI tool or via your internal translator introduces additional errors into the test verification process that will only cause expanded testing delays, timelines being missed and a misconception that your systems have been fully tested. I have also heard phrases like "my translator or software is HIPAA compliant, so we are okay". All I can do is wonder why people believe all these misconceptions. "If you tell a lie long enough, it will still never be considered a fact" Allow me to digress on this subject for people without formal QA training and expertise. A test file is derived from a test case which is derived from a set of test conditions which is created in response to testing and proving a business requirement. The test case must have an expected result in order to prove proper functionality. Free test files do not have expected results and in essence would have to be retrofitted into the test conditions. Using a free file causes test analysts to decompose the file to determine all of the pertinent contents, then adjusting the test file to meet test cases and conditions. A very important point is that test files, test cases and test conditions must have traceability to be usable, otherwise the test team has to look everywhere for defect correction when a discrepancy is found. Is the defect in the translator, the EDI validation tool, in the test file itself, in the understanding of the guides, was the business requirement written correctly, etc...? In reality, available free test files are only an infinitesimal representation of all the conditions that need to be tested to correctly ascertain whether or not business functionality has been proven. I am not sure there is great benefit in providing a test analyst 1% - 5% of their overall test file needs. There was an estimate of 1400 possible test files for Oxygen Therapy (both positive and negative test conditions), care to guess how many organizations executed even one quarter of the possible files? QA personnel should know better than to assume any particular aspect or product is 100% correct. Now, let's look at the bigger picture which will cause even more contention. If you offer free test files, through which validation engine where they executed? EDI validation tools all give different answers, but which one is right? All of them? None of them? A combination of all of them? Lets examine a real scenario: Payer X using EDI tool A to verify their files, Clearinghouse Y uses EDI tool B to verify their files, Providers X,Y and Z all use different EDI tools for their validation. All covered entities believe they are compliant because their EDI tool told them so. Provider Y sends what they believe to be a complaint file to Payer X, yet the transaction is rejected by Payer X because their front end EDI tool says it is not compliant, and so on and so forth... Can you see what the industry will uncover, but by that time it is too late. There is also discussion about having all the EDI vendors perform testing with each other, why? Shouldn't they have tested their product completely before they released it to the industry? If a company produces a quality product they should be rewarded for doing so, and quality usually comes with a price. Last I checked, this is still a free market society. This should also prove the point that EDI validation is just a small part of the information chain and cannot be seen as a so-called verification or certification point. The same goes for translators as well. These are all various pieces making up a whole and I highly doubt anyone of them are defect free. Covered Entities MUST examine testing from a business process and business functionality perspective. So many of us with years in quality assurance agree that you cannot stress enough that you must start from the known and progress to the unknown, not from unknown to unknown and NEVER assume any piece of software is 100% defect free. Therefore you should have test conditions that represent application test data, specific translator test data, EDI validation test data, etc. If you start with trying to prove business functionality in conjunction with a very robust test effort, you will uncover translator and EDI validation defects. If you start from EDI validation and go backwards, you will not find all your defects and will leave your organization exposed. How does an organization know that their systems have been fully tested? Certainly not by running free test files through their application systems, although there may very well be people doing this. By offering free files you open yourself up to many, many issues concerning viability, traceability and responsibility. Is this really the best use of volunteer time? I don't believe it is. I fear that true testing methodologies, concepts and best practices are not being promulgated enough in this industry and in this forum and organizations will have a false sense of security about their applications systems. I strongly believe that people need to really open their eyes and see what is happening in this industry and not assume everything will be taken care of for them. There are many organizations looking for help to comply with HIPAA transactions and code sets, and they look to us for assistance, we must try to not get caught up with our own business initiatives and do what it right for the industry as a whole, isn't that the definition of volunteerism, to help the greater good? This in no way reflects on the all the volunteers that have graciously given their time to work for the betterment of the industry, but a much more realistic and hard-hitting approach needs to be taken with a focus on true implementation if we are to expect this venture to be a successful one. I apologize for leaving off the sugar coating, but their needs to be a line drawn in the sand not a hole to bury one's head. This only reflects the tip of the iceberg and the frustration currently being voiced on the current testing implementation shortcomings will only grow and are very much welcome to assist in driving the point home. New education and industry initiatives are needed... If I may share a few words of wisdom: "Failure is not a single, cataclysmic event. You don't fail overnight. Instead, failure is a few errors in judgment, repeated every day." -- Jim Rohn "The quality of the solution you pick will be in direct proportion to the quantity of the solutions you consider." -- Brian Tracy Mark A Lott Please note that it may take up to 72 hours to process your request. <P>The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. Posting of advertisements or other commercial use of this listserv is specifically prohibited. Please note that it may take up to 72 hours to process your request. <P>The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. Posting of advertisements or other commercial use of this listserv is specifically prohibited.
