12Oct2009 (UTC +8) On Mon, Oct 12, 2009 at 13:54, Oscar Plameras <[email protected]> wrote: > I think it's silly to spend so much money and time to test the > Election System by reviewing Source code.
Fair enough opinion. This statement makes sense *if* only source code review is made, *without* acquiring other audit evidence as well. By other audit evidence I mean getting to evaluate the entire software development life cycle of the IT product and how it is implemented, as well as user and administrator documentation, packaging, and inspecting the IT product developer's production and R&D offices. Getting test results from test labs is also part of what constitute audit evidence. > From my experience, end users implement acceptance testing of the > system by developing a series of test > other than source code review.The main idea is to simulate scenarios > of operations with input test data > and pre-defining the expected results. Several scenarios are covered > with the input data that's prepared. > > The Election system itself is a simple count and tabulate system and > that is not difficult to simulate. Dynamic code analysis, which is what you describe above, is just a subset of the test process in a evaluation process. By the way, UAT is *not* included of an evaluation and assurance process. Dynamic code testing is great because the evaluator, or auditor, gets to see how the IT product runs in relation to other components, which is difficult to simulate on paper alone. It has many limitations though, which testing is limited to *known* issues only, or calculating for run-to-run totals, or similar output. By itself, it is inadequate to provide assurance. But without it, test evidence is incomplete. > Hardly no commercial developer will allow third parties to have source > code access to their propriety > software. And in general, commercial confidence protects the privacy > of these codes.under the trade > secrets act of countries. I think the Philippines is a signatory to that. There are hundreds of proprietary software out there that had their software evaluated by 3rd parties. To name a few, MS Windows 2003 Server (evaluated by a friend of mine), VMware ESX, evaluated by another colleague of mine who I'll be meeting in a few days to discuss work stuff --and he's done IBM Websphere too. Check Point FireWall-1 / VPN-1 and other products, the evaluation led by another friend of mine, and I was part of his team back in the 1990's and early 2000's. Then there's Solaris too, RIM's Blackberry, and many more. None of them are complaining that their Intellectual Property were compromised. > And lastly, which source codes are they going to review. The > application source codes? But application > source codes interacts with system source codes. Are they going to > review system source codes, too? > What about the source codes of all firmware chips used in the system? > Are they goind to review those source codes, > too? How long is a piece of string? The code done by one programmer > maybe anathema to another and so > source code review leads to more controversies. As you know > programmers are full of egos and one argument > leads to another and another. The point is if it does the defined > specifications, it does not matter how or why the > code is written that way. > > Reviewing source codes is a mine field of difficult issues to deal with. It is not easy, I agree. But not impossible. Been there, done that, and I've got t-shirts to prove it ;) > The simplest and easieast is to test by outcome, not how the code and > why the code is written that > way. After all, we are interested in the integrity of the system not > the integrity of the code. Yes, all of us Filipinos are interested in the integrity of the outcome of the 2010 National Elections. But what we are all saying here, is that simply testing the interfaces of the AES is *not* enough to give the Filipino people the assurance that the AES will work as *exactly* expected --no more, no less. I read the other responses here, and they're also saying that external testing of the AES will not be meaningful enough. Drexx Laggui -- CISA, CISSP, CFE Associate, ISO27001 LA, CCSI, CSA http://www.laggui.com ( Singapore / Manila / California ) Computer forensics; Penetration testing; QMS & ISMS developers; K-Transfer PGP fingerprint = 6E62 A089 E3EA 1B93 BFB4 8363 FFEC 3976 FF31 8A4E _________________________________________________ Philippine Linux Users' Group (PLUG) Mailing List http://lists.linux.org.ph/mailman/listinfo/plug Searchable Archives: http://archives.free.net.ph
