http://www.ieee-pses.org/symposium http://www.emc2004.org/ --------------------------------------------------
Ed As you have changed the Subject string, I am allowed to have another "final word"... Your questions are good and fundamental to the field of hardware and software co-verification and validation. And I do not have generic answers that could be considered correct. But, I have a few (half-cooked) ideas... "Traceability" is relatively (hah!) easy; control the release and use of software systems using CVS. But "proving confidence" through V&V has grown a whole industry, and provided fodder for many a PhD thesis. In any case, it is important to treat s/w as a "natural language" expression of mathematical equations. That is, have a firm grasp of a finite mathematical model of the data acquisition / data analysis system before any application is loaded, or a single line of code is written. Do not attempt to prove correct functionality of the whole platform. Limit formal proofs to singular processes; and provide a formal test plan for each individual process to the auditor. And you are correct that you will eventually have to "just accept that the pencil is whatever it is" (I like that analogy, can I quote?). The granularity of formal proofs and associated tests will be based on the requirements of the end-user and the applicable standards (which may be ambiguous). Modern software development emphasizes using the highest level of abstraction available (mostly for development productivity). But higher levels of abstractions increase (geometrically) the number of paths that code can follow. So if the complexity (and non-determinacy) of a GUI is removed, and the execution environment is limited to a CLI (aka "shell"), the requirements for formal proof are significantly reduced. Finally, ignore the academics and their games with stochastic methods. As for formal definitions, I am NOT a V&V expert, so I am going to take the coward's way out... and humbly suggest the following refs: FDA Guidance Document 2 FDA Guidance Document 4 IEC 60880 (extreme, but it's for nukes) IEC 12207 (overall look at the mess w/the misnomer "s/w development") IEEE 1012 IEEE 1059 (does not enlighten one of V & V, but has formal process definitions) http://www.fda.gov/cdrh/comp/guidance/938.pdf http://www.a2la.org/guidance/adequate_for_use.pdf ftp://ftp.fasor.com/pub/iso25/validation/ MISRA C SPARK Ada LCLint/Splint luck to all who dare to verify and validate software, Brian -----Original Message----- From: Price, Ed [ mailto:[email protected]] Sent: Tuesday, August 17, 2004 4:12 PM To: [email protected] Subject: RE: Which Operating System - Now Traceability From: Brian O'Connell [ mailto:[email protected]] Sent: Tuesday, August 17, 2004 1:00 PM To: [email protected]; [email protected] Subject: RE: Which Operating System - 2000 vs XP to use? Good People I got wayyyyy too much OS theory in school (was a CS major). SNIP My final word on this subject (honest...). If your lab is accredited to IEC 17025, and the auditor requires software verification, validation, and traceability for instrument control and/or numerics; then you must have access to, or knowledge of, system internals. The most simple solution, for people whose job/hobby is not systems administration, is to control the instruments using command-line Linux (or DOS if you do not need USB or other modern I/O) on a dedicated box, and export the data to an appropriate machine loaded with your number cruncher/graphing apps, typically an XP box. Stand-alone numerics is easilly tested using "black-box" and associated regression tests. Important note: the (US) FDA will always require full validaton. luck, Brian Brian: Could you please explain (in beginner's terms) what are the concerns regarding "validation" of the software used in a data acquisition system or an OS. About a decade ago, I had gotten just a hint of exposure to software control at General Dynamics. At that time, my area of GD typically used HP "desktop controllers" like the HP-9836, running HP Unix or Pascal or Basic. We used data acquisition applications that were either self-authored or purchased >from HP. GD's QA people had required that all software be registered with them, and that working copies be periodically audited for changes by performance of a digital checksum on the file. The HP software was proprietary, and we couldn't see anything about how the internals worked. The QA control was only concerned with "creeping code shift"; that is, undocumented changes made on the working copy of the code. As far as I know, they never tried investigate any deeper. I validated the operation of the HP data acquisition code by using the technique, now described in MIL-STD-461E, of injecting known signal levels into the probe end of the coax cable and seeing if the acquisition system would output the correct frequency and amplitude data. (I did the same for the self-authored code, but this was very redundant, since that code had been very rigorously verified as module by module went together.) Regardless, with a few hours time, I could feel very certain that the code was yielding correct results. But your comments about "validation" give me a chill suspicion that I was blissfully ignorant of something important, something I still don't see in hindsight! Do current auditing standards require far more samples to justify confidence? How many samples would you need? So what is involved with proving confidence in an OS? I know that HP simply stated that their (acquisition) code was proprietary, and that we would just have to trust them that everything was as it should be. And nobody ever thought to question their OS. But what would you have to do to validate an OS? Are you saying that confidence requires knowing that an OS yields the same result for 10^9 square-root operations? Or is 10^15 needed? Does every algorithm need to be examined and proved? Why would using a CLI be any more trustworthy than using a GUI? Using these standards, could we even prove confidence in a hand-held calculator? What about the firmware in a printer? How confident are we that it will always print "47 dBuV" every time we send that data to it? How far does traceability need to go? Do you need to document the origin and chemical composition of the graphite in a pencil, or do you just accept that the pencil is whatever it is, so long as it makes marks you can read? As you can see, I'm groping at the concepts and definitions. Regards, Ed ------------------------------------------- This message is from the IEEE Product Safety Engineering Society emc-pstc discussion list. IEEE PSES Main Website: http://www.ieee-pses.org/ To post a message send your e-mail to [email protected] Instructions for use of the list server: http://listserv.ieee.org/listserv/request/user-guide.html List rules: http://www.ieee-pses.org/listrules.html For help, send mail to the list administrators: Ron Pickard: [email protected] Dave Heald: [email protected] For policy questions, send mail to: Richard Nute: [email protected] Jim Bacher: [email protected] All emc-pstc postings are archived and searchable on the web at: http://www.ieeecommunities.org/emc-pstc
