RE: VALIDATION or Certification

[Mark A Lott]

Margaret,   Very well stated!  This debate and confusion is growing more everyday and it was really only a matter of time because the definition of certification that was recommended to an entire industry was vague and incomplete in regards to industry accepted norms.   If I may add to the discussion with excerpts from published national standard setting organizations and processes used in almost every other industry that incorporate certification into their respective markets.  I have added my comments below with ***   What is an appropriate testing method for conformance certification? The test method must be adequate and appropriate within the conformance testing and certification program in which it is used.  Beyond these properties, test methods (and thus the tests) should be objective, have adequate coverage, and correctly implement the specification.  In trying to meet these requirements, those using and applying the test method should not make the common mistake of allowing the test method to become the ‘specification’.  This means that users will build the transactions to pass the conformance tests of a specific tool, rather than building to the specification.  ***This challenge is exactly what is happening today, organizations are testing transactions against vendor’s software instead of the specification themselves. This is a major issue that will have adverse consequences if it is not corrected. A covered entity must create and utilize test cases that test all aspects of the implementation, the translator, the application system and the EDI validation tool. All three represent software instances with known and unknown defects, you must build test cases that flush out all the errors in every part of your environment. An objective test method allows for test results to be reproducible and repeatable and is especially testing tool independent.   What is a conformance test and how does it work? A conformance test suite is a collection of combinations of valid and invalid inputs to the implementation being tested, together with a corresponding collection of expected results. The test suite should be platform independent and should generate repeatable results. The conformance testing program should strive to be as impartial and objective as possible, which means that you must remove the subjectivity of both the testing procedures and the testing tools being used.   ***The test methods and test suites are designed to uncover errors in all HIPAA software solutions including translators, vendor software, EDI validation tools and your internal applications.   Why can’t conformance certification be performed by software alone? First and foremost conformance certification is a process not a product. When using a software only solution you will find that it is highly unlikely that software products are free from defects and thus cannot be can be deemed 100% accurate as to compliance. When submitting the same transaction through the different software engines, covered entities will receive differing analyses of their transaction compliance. What comfort level can be obtained if you don’t know what is the correct and agreed upon benchmark for certification. This is precisely why certification is a process of quality assurance and cannot be performed by software analysis alone.    What are certification programs? Certification programs are communication tools designed to reduce the cost of exchanging information between buyer and seller. The quality of the information conveyed depends on both the impartiality and competence of the certifier and the adequacy and appropriateness of the tests against which the product is evaluated. Certification may result in widespread buyer deception if the performance characteristics or test methods contained in a standard are insufficient to assure adequate product performance or if the buyer is misinformed as to the extent to which certification characteristics have been evaluated.   *** Evaluation criteria should include artifacts such as testing methodology, test plans, test scripts, test files, code coverage (adequate quantity and quality of test files to exercise changed code), regression testing and interoperability testing (the ability to send and receive certified files that are portable across trading partner communities). The United States has some of the most intricate certification schemes in the world. Federal, state, and local government certification activities impact on almost every aspect of life in the United States. It is therefore important for all buyers to understand the certification process and what particular marks of conformity mean to enable them to assess the value of certification information to make intelligent choices.   What specific recommendations can you base certification programs on? The ABC’s of the U.S. Conformity Assessment System, NISTIR 6014,April 1997, The U.S. Certification System from a Government Perspective, NISTIR 6077,October, 1997 Developing Federal Standards and Accreditations for Data Protection Products, October, 1995 Standardization and Related Activities: General Vocabulary ISO/IEC Guide 2: 1996, General Requirements for the Competence of Testing Laboratories. ISO/IEC Guide 25:1990, Procedures and Requirements, NIST Handbook 150, March 1994. American National Standards for Certification - Third Party Certification Program ANSI Z- 34.1-1987 UL Standard for Safety for Software in Programmable Components. UL 1998, 2nd ed   Regards,   Mark A Lott President/CEO HIPAA Testing, Inc. www.hipaatesting.com   Executive Co-Chair HCCO - Transactions Group HIPAA Conformance Certification Organization www.hcco.us   Office: 480-946-7200 Cell:   480-580-4415 Fax:  877-825-8309   CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message     --- The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. These listservs should not be used for commercial marketing purposes or discussion of specific vendor products and services. They also are not intended to be used as a forum for personal disagreements or unprofessional communication at any time. You are currently subscribed to wedi-testing as: archive@jab.org To unsubscribe from this list, go to the Subscribe/Unsubscribe form at http://subscribe.wedi.org or send a blank email to [EMAIL PROTECTED] If you need to unsubscribe but your current email address is not the same as the address subscribed to the list, please use the Subscribe/Unsubscribe form at http://subscribe.wedi.org CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. --- The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. These listservs should not be used for commercial marketing purposes or discussion of specific vendor products and services. They also are not intended to be used as a forum for personal disagreements or unprofessional communication at any time. You are currently subscribed to wedi-testing as: archive@jab.org To unsubscribe from this list, go to the Subscribe/Unsubscribe form at http://subscribe.wedi.org or send a blank email to [EMAIL PROTECTED] If you need to unsubscribe but your current email address is not the same as the address subscribed to the list, please use the Subscribe/Unsubscribe form at http://subscribe.wedi.org