-Caveat Lector- Subject: House Testimony of David C. Hall, Senior Consultant, CARA Corporation > > HOUSE COMMITTEE ON GOVERNMENT > REFORM AND OVERSIGHT > Subcommittee on Government Management, Information and Technology > Field Hearing on Year 2000 Efforts > September 3, 1998 > Chicago, Illinois > Testimony of David C. Hall, Senior Consultant, CARA Corporation > > > Subject: Infrastructure and Embedded Systems Year 2000 Efforts > > Today I will limit my testimony to a few statements that, in my >opinion, describe the state > of the Infrastructure and Embedded Systems efforts to date. The >written testimony > provides some additional information. The statements are as follows: > > 1. Fewer than 10% of the enterprises in the world have begun >serious embedded > systems and equipment testing. > > 2. The test results thus far agree with the original estimates of >the magnitude of the > embedded systems problem-that anywhere from 1% to 15% of the >microprocessors-based > embedded systems and items of equipment exhibit some type of Year 2000 >impact. These > impacts range from minor to catastrophic. > > 3. These impacts are showing up on an individual basis, that is, >by serial number rather > than by model number. > > 4. This means that every microprocessor-based embedded system and >equipment > item must be individually tested to be sure of its Year 2000 status > > 5. There is insufficient time and trained resources to test every >microprocessor-based > embedded system and equipment item in the United States, much less >every one in the > world. > > 6. Therefore, we must assume that not all Year 2000 impacts will >be found > and fixed by January 1, 2000, and should expect disruptions and >disturbances from Year > 2000 embedded systems and equipment impacts. > > 7. How many disruptions and disturbances and how serious they are >will depend > entirely upon how much work is accomplished, both in remediation and >contingency > planning, before January 1, 2000. > 8. Since we cannot find and fix all the impacts, we should make a >concerted effort to > find and fix those that pose the most serious risks to public health, >safety and the > environment. > > > Global Implications of the Year 2000 Embedded Systems > Problem > By: > David C. Hall > Senior Consultant > Year 2000 Infrastructure and Embedded Systems Engineering > CARA Corporation > August 1998 > > The part of the Year 2000 Problem caused by embedded systems >(anything > containing a microprocessor of microcontroller) will be the most >extensive and expensive > part of the Year 2000 Problem. The world has spent the better part of >five decades > automating, networking, centralizing, and integrating our facilities, >plants, public and private > infrastructures, communications, financial systems, health systems, and >just about > everything else. With over 10 billion microprocessors sold worldwide >in the last five years > alone, we have a tremendous job of finding, testing and fixing to >accomplish in less than > two years. The bottom line to this > problem is that wherever on or in the globe there is an electronic >device, there may be Year > 2000 problems and risk. > > The Whole Issue > The Year 2000 Problem was originally seen as just an Information >Technology (IT) problem, > affecting only the old legacy mainframe software programs. However, >over the past few > years it has become known as a Business Problem, because it affects the >reliability of all > computer-based systems and equipment used, be they Information >Technology systems, > desktop computing systems, or facility environmental control systems. >In each of these > cases, the reliability of information is the most important >consideration about Year 2000 > problems. > In human terms, bad information may cause us to make a wrong decision. >In machine > terms, bad information may cause a malfunction, a total shutdown, or >produce the wrong > result. The difference is that humans can often interpret bad >information, through > application of intelligence, and compensate. Machines cannot >compensate, they will do > exactly what they have been programmed to do. > > As a country, community, or person, we are part of a global >neighborhood that has come to > depend upon many different types of computers to process data, manage >information, > communicate, and control all types of processes. Our modern society >depends upon > these computers for every facet of our social structure. They control >and operate our > facilities, plants, hospitals, finances, traffic lights, electrical >power generation and > transmission, water and sewage plants-everything we now depend upon, >globally. > > Fixing this problem has become a much larger problem because we have >less than two > years to correct 40+ years of putting "bad" logic in every type of >computer in the world. > The completion date we must meet cannot be slipped or changed. January >1, 2000 is fixed > and it will arrive exactly as predicted. In addition, there are no >rules and no standards that > were > followed by the people putting the bad logic in those billions of >computers. Nothing was > done to make it easy to find and fix the numerous problems. Each >solution requires time > and trained personnel, two ingredients we lack. > > Effect on Embedded Systems > There are an estimated 40 billion microprocessor-related chips in >service around the world. > At the low end they are very simple, such as timers with a capability >of counting seconds > or minutes one by one until it receives a stop signal. At the high >end, they are fully > functional computers on a chip, which perform sophisticated tasks. To >most of us, these > things aren't "real computers." No keyboard, no monitor, nothing. > The terminology people use for these things varies; embedded chips, > non-compute-devices, and black boxes. From the early days of >computing, these chips > have been the provinces of the engineers, not the programmers. > As we began the search for potential problems that will occur after, >on, or before January 1, > 2000, a number of "non-compute" devices were immediately identified: >fax machines that > will print the wrong date at the top of the page, telephone switches >that won't work, hospital > equipment that will malfunction, and electric power generation systems >with date functions. > > Such problems turned out to rule rather than the exception. There are >embedded chips > everywhere. They are in our air conditioners and furnaces, >automobiles, elevators, alarms, > traffic lights, aircraft, ships, controlling our manufacturing plants >and our electric power > plants, etc. Rumors and speculation have sprung up about how >everything was going to > quit come the Year 2000. That is not he whole truth, but there have >been > sufficient problems identified in individual systems and equipment that >I believe that we > must test everything to determine whether or not there is a problem in >that specific item of > equipment or system. In tests accomplished so far, anywhere from 1% to >10% of an > enterprise's systems and equipment items exhibit Year 2000 impacts. >How does one find > out whether or not their systems and equipment will have Year 2000 >impacts? We can't go > to a keyboard and test every chip with a Year 2000 rollover program. >We > can't list the code on a screen. In many cases no one knows exactly >what types of chips > have been used in their systems or purchased equipment. Adding to this >uncertainty are > the vendors and manufacturers, who originally gave a blank stare when >asked about Year > 2000 compliance, and then stated with great assurance that "It isn't a >problem with our > product." And then retracted that statement after a few tests. > > The basic fact to understand and hold onto is that microprocessor-based >systems and > equipment underpin most of the world's social, manufacturing and >engineering base. The > world's energy supplies (oil, gas, coal, and nuclear) depend upon >embedded systems. > Planes fly, and ship sail, based on embedded systems. Industries used >embedded > systems to produce the world's supplies of everything from drugs to >hats to more > computers. Buildings use embedded systems to control their >environment, lighting and > security. The food we eat and the water we drink primarily come from >processes that > depend upon embedded systems, as does car manufacturer and railway/air >traffic control > systems, and communications, and so on. But what's the problem? >Haven't we always > had problems in such systems and been able to fix them once they >failed? Yes, but we > have not had all of these systems potentially affected at the same >time, all over the globe. > Sure, you can fix a few problems, here and there. How do you fix >thousands or millions > that happen at the same time? > > So embedded systems are the prime components, the actual foundation, of >our > global infrastructure. They are also the commercial building blocks of >engineering and > manufacturing worldwide. Addressing the potential Year 2000 problems >for these systems > is at least as important as doing it for banking and financial >institutions. In my opinion it is > much more important. And fixing the problem is embedded systems is far >more > complicated and expensive. All of the problems that exist in the >traditional big mainframe > systems also exist in embedded systems. We have problems arising at >the processor > level, from operating systems, from packages/applications, and from >custom-built systems. > > The technical solutions to any problems found are also the same-some >replacements, > some modifications, and some workarounds. > > But the big difference is time required to find out whether or not a >specific system > or item of equipment has a Year 2000 problem, which normally will >require a very > specific test. Since it is not possible to type test any system or >item of equipment > containing microprocessors, we have to assess and possibly test every >individual system > and item of equipment to be sure we have caught all Year 2000 problems. >Once we have > found the real Year 2000 problems, the time necessary to remediate and >implement Year > 2000 solutions can be extremely long. Embedded systems can be very >complex, and > many of them are used, and many of them are used to control or monitor >some very > high-value facilities and processes. A large building or complex of >buildings may have 10 or > 20 embedded systems interfaced together, controlling everything from >the lighting to the > elevators. A large facility such as a petrochemical plant or oil >platform at sea will have > hundreds or thousands of embedded systems all interfaced together. In >both cases, the > embedded systems have been bought for different reasons by different >people over the > people over the years, usually mirroring the gradual development of the >facility, The > continued operation of the facility or the plant is now dependent upon >the successful > continuous operation of the embedded systems. > > Because the facilities and plants are so valuable, production managers >and > engineering staff are justifiably fearful of failures. When embedded >systems fail, high-value > processes shut down, and costs of unexpected shutdown can be hundreds >of thousands of > dollars. Even for small manufacturing companies or facility owners, >the cost are crucial, > because the continued operation of the facility or plant is their only >source of income. The > pressure to keep the systems running is great. The result is that the >world has a huge > number of aging embedded systems, based on languages, applications, and >processors for > which the necessary skills, personnel, and vendors are gone. Embedded >systems are > more difficult to inventory, because some of them are so old that the >documentation has > literally been lost or discarded. Systems dating from the early 1970s >through the early > 1980s are common. They are also much harder to get at, since many of >them are located > in the walls, under the floors, in harsh environments, etc. Doing >triage planning is very > complicated, because there is a risk that taking the systems through a >mock Year 2000 > change will cause the operation to fail, with huge cost penalties. >Applying a fix is also a > problem, again because of the huge potential to cause an unexpected >failure. > > So to fix the Year 2000 problems in embedded systems, you need people >who > understand embedded systems technology, the production processes you >are > working with, and the commercial impact of mistakes in a manufacturing >environment. And > you need people who can travel to many different places and work under >harsh > environments such as the seabed, in nuclear plants, etc. There are not >enough such > trained, capable people in the world to find out whether our systems >and equipment really > do have Year 2000 problems and then fix the ones that are found between >now and 2000. > > Other areas where embedded systems can impact an enterprise is in the >transportation > systems and supplier/customer chain. In terms of the transportation >systems, if the > over-the-road carrier, airline or ship you depend upon to bring in >supplies and carry away > goods ceases operations for weeks or months, what would you do? Even >if you have > another carrier you could use, is he going to be available at a price >you can afford? With > the example of the U.S. United Parcel Service strike and the resulting >problems of trying to > get alternative means of transportation last year, I would submit that >you would not easily > or quickly be able to obtain such alternative means. Especially if >everyone else is trying to > do the same thing at the same time. Here again is the global aspect of >the Year 2000 > Problem. Everyone else will be trying to do the same thing at the same >time around the > globe. > > In evaluating the results of embedded systems and equipment test thus >far, the major > implication is that we cannot type test. This basically means that you >must test each > individual system and item of equipment in its operational environment >to be assured of a > correct result. We cannot test 10 of one model number and then assume >that all ten > thousand will react to Year 2000 in the same way. There has never been >a standard or > specification for how to build Year 2000 compliant components, so each >manufacturer and > vendor has been able to use "their" way or to ignore it altogether. At >the component level, > each system and item of equipment can be fundamentally different as far >as reaction to > Year 2000 is concerned. So to know what will happen when Year 2000 >dates are input, > one must test each item individually. > > Year 2000 Embedded Systems "Best Practices" > The method that have been developed to deal with this Problem over the >past > three years mirror those being used for Information Technology systems >and > programs: > > 1. Awareness > 2. Inventory and evaluation > 3. Remediation > 4. Testing > 5. Implementation > 6. Monitoring > > However, the persuasiveness of embedded systems makes this project >extremely complex > and management intensive. It is by far the most complex risk >management project any > enterprise has undertaken. The requirement to determine Year 2000 >compliance by serial > number for each system and item of > equipment means that an enterprise must undertake and complete a more >complete > inventory than it has ever required. This inventory must contain all >items and systems > using electricity and be complete down to the chip set hardware and >firmware versions. To > fully evaluate each item and system, specific tests must be >accomplished according to > formal test procedures (some experts note that up to 30 test per system >may be required > to fully map out the functionality of the system.) The possibilities >of false negatives and > false positives, as well as the possibility of damaging the equipment >during the tests, must > be considered and allowed for. A further problem is that these tests >must be conducted on > systems and equipment that are actively engaged in controlling and >monitoring the > operational processes of our plants and facilities. I fully expect >that more resources will be > expended in determining that you do not have > problems than in fixing the problems you find. However, you must test >to know where your > problems are and what the impacts will be. You cannot fix or plan for >problems you don't > know you have. > > Remediation of the problem found entails several complex areas. First >is > determining the most economic way of fixing the system(s) or item(s) of >equipment. This > usually requires extensive discussions with the vendor, if they are >still available. If it is > necessary to change the hardware and/or firmware, finding a vendor to >design and develop > an appropriate fix is currently difficult, and will get more difficult. >The resources of the > better vendors are becoming strained as more and more customers request >work. All of > the better vendors have sold equipment and systems on a worldwide >basis, and all former > and current customers will be requesting help. > > Testing the fixes also entails several problems. Systems and equipment >must be tested in > their operational environment to be fully assured of a successful >process. Testing a fix > requires the same methodology and procedures used in acceptance >testing. This may > have to wait until a plant or facility is down for maintenance or >repairs. The tests > themselves may damage a plant or facility if there are "bugs" in the >remediated code or > hardware. As in Information Technology testing, the more rigorous the >testing, the more > successful the result. But in this case, it will be very difficult to >build a separate testing > environment, and we are working with very expensive equipment. > > Implementing a successful fix in the appropriate production environment >will also be > expensive and difficult since we will have to bring down our >operational processes and > systems to install and test the fix. Failures at this point will be >very costly both in terms of > resources and operational capability. > > Monitoring after a successful fix is of extreme importance, as it will >be very easy for an > unaware user to update, maintain, or repair a compliant system or item >of equipment into > noncompliance. We have never had to monitor ALL of our equipment and >systems down to > the chip set level before. Very few companies or agencies have in- >place procedures and > controls necessary to accomplish this effectively. > > Since, in my opinion, there is no possibility that all embedded systems >impacts will be > found and remediated before their failure horizons, contingency >planning has become a high > priority. All enterprises and agencies must develop and test >contingency planes for > multiple failures of both internal and external systems and equipment. >A plausible > contingency planning scenario is to prepare for up to two weeks of >intermittent utilities, with > several months of intermittent supply deliveries. However, each >enterprise should > determine their own planning scenario, since that will determine >resource use and planning. > > Summary > >From what I can determine at this time, few companies or government >agencies in any > country of the world have yet to recognize or attempted to find out the >scale of their > potential Year 2000 embedded systems problem. Systems and equipment >items are not > yet failing, because embedded systems and equipment tend to have a >lookahead time of > less than one month. So the failures will come in 1999 and 2000. >Nonetheless, from my > work over the past three years in this area and from tests conducted by >numerous firms, > we know that the likelihood of failure or erroneous data generation of >numbers of embedded > systems and equipment is high. You cannot fix or plan for problems >that you do not know > about. The comparatively few companies and government agencies that >are in the > vanguard of Year 2000 work will be ready. Other large companies and >Government > agencies may be able to fix their problems by throwing lots of money at >them, though > people resources are currently very scarce and will get scarcer as time >passes. The > small and medium size companies that make up the bulk of jobs in any >country are > in trouble. The state and local governments who supply the bulk of >public services > to a majority of citizens are in trouble. > > Will we be able to mitigate this coming crisis? In my opinion the >answer is Yes, with the > following caveat Companies, government agencies and the general public >must become > more aware of the potential problem and must assign more resources to >addressing it. If > everyone waits to start to work on this issue until "next year or next >month," then we are > allowing our problems to escalate. We are already at the point that >not all problems will be > able to be found and fixed before their failure dates. Contingency >planes must be > developed and put in place for the expected failures. To wait any >longer before working on > this problem multiplies the risk of large-scale disruptions of our >basic global infrastructures, > be they economic, public, or governmental. > > > > Author: David C. Hall > Senior Consultant > Infrastructure and Embedded Systems Engineering > CARA Corporation > 1900 Spring Road, Suite 450 > Oakbrook, IL 60523 > > Phone-630-368-2823 > Fax-630-368-2800 >http://home.swbell.net/adheath/dhall.htm >) *** DECLARATION & DISCLAIMER ========== CTRL is a discussion and informational exchange list. Proselyzting propagandic screeds are not allowed. Substance—not soapboxing! These are sordid matters and 'conspiracy theory', with its many half-truths, misdirections and outright frauds is used politically by different groups with major and minor effects spread throughout the spectrum of time and thought. That being said, CTRL gives no endorsement to the validity of posts, and always suggests to readers; be wary of what you read. CTRL gives no credeence to Holocaust denial and nazi's need not apply. Let us please be civil and as always, Caveat Lector. ======================================================================== To subscribe to Conspiracy Theory Research List[CTRL] send email: SUBSCRIBE CTRL [to:] [EMAIL PROTECTED] To UNsubscribe to Conspiracy Theory Research List[CTRL] send email: SIGNOFF CTRL [to:] [EMAIL PROTECTED] Om
