Kirk: I agree that rewriting code, and the associated testing is very expensive for large critical system like medical software. You are right about the cost being in excess of $100k (easily). With these kinds of cost why would I want to incur these costs. The description is long so you may want to skip the details and just go to the executive summary - at the bottom.
1. Old code can be crippling: Our application started out with version 2.0.10 (pre server) of 4D. If we did not rewrite code we would have had a very crippled multiuser environment, and have old code that did not take advantage of the better implementation features of newer versions of 4D. We never could have had 120 users on our 4D version 3.x of 4D without a major rewrite. As sites got terabytes of data and used the solution for all their work the demand on the system performance became greater each year. 2. Many of our competitors in the market sold applications that were very long in the tooth. It showed, and it was easy to get new customers interested in our products because we refactored our code. Their owners make more money though because they do not incur these costs. They will not survive in the long haul. 3. I had the attitude of adding many features every year, and keeping up with the UI demands of potential new customers. Government intrusion into the business demanded features that were often stupid and never used, but were required to 'play' in the market. To be fair they had good ideas too, but they got them all from the vendor community that they then passed along to other vendors as requirements. As one of the first players in our market we already had them implemented Let me explain the way we accomplished this. With an installed base of over 4,000 concurrent users at about 200 sites, and add to that the randomness of over 20,000 people using the software one must tread very carefully when modifying code. Medical people have little tolerance for crippling, or error prone code (I appreciate this as a patient). I know there are OEMs out there with larger installed sites, and concurrent users so they too know these issues (like Kirk). With this installed base as well as government intrusion in the marketplace we just could not upgrade our solution with each new version of 4D. This was not a fault of 4D Inc, but the resources required for testing in multiple jurisdictions each year. We had to have people working on validation testing almost all the time. This cost is not included in my estimate of the cost of refactoring. With a version of 4D that I decided to rewrite our shell (as I am doing now with v15) we start from no code and no tables. We have a list of features we want to implement both from the perspective of new features in 4D, new features in our code, and better ways to implement something we already have done. With this in mind the new shell is written. He last time I did this it took me 4 wonderful months to accomplish. As a programmer those are fun months. We knew what the feature set we were writing to ahead of time. Therefore we could write it better. We knew where we had clumsy code and could write the features in an elegant, obvious, easier to maintain manner. For example when Object Tools came out that was a big part of one rewrite, but freeing. Same for Interprocess Variables, 4D Server. There are many other over the years. The advantage was we could rewrite the code knowing the feature set we required to accommodate that the code we wrote 15 or 26 years earlier we never had many of them in mind. This is the attitude and objective of our rewrite of the base shell. This typically includes security (very nice to have in the base shell), interface, online help, debugging tools, standard interfaces, administration, user handling, network testing, performance testing, automated reporting from the client to us, to name just a few. We would then use this new Shell as to starter to rewrite a small application that we used internally to process medical payments of several million dollars a week. Sounds scary but with error code detection we knew if it worked well or not. It is a small application doing a big job. With this application we got to discover some of the bugs and fix them. This typically took about a month of developer time. We then had a piece of software that we used to run our whole business - JERMS. This one would take much longer to rewrite - about 6 months. This software was used by all our staff. Nice thing is that it was all in house users hammering away at the server all day long. In the last 10 years clients accessed this system too. The interface was the same was as for our medical application. This we worked through with for another six months to resolve issues. We would learn a lot about our refactored code and the new version of 4D through this experience. Then the task of refactoring our main product. Before government intrusion into the market we could go at our own pace so we could manage this better. Even with only one government intrusion we could easily manage the process as there was a working relationship between the twenty vendors and the government as to the pace of required change. There is a whole different story when many more government's got into the demand game. We would decide what of the features we would refactor in code and of course implement the required new features by clients or government. A must of this was having two development servers. We called one RDev (Release Dev) which would only get bug fixes. This way we could quickly deliver a bug fix without worrying about code changes in a multitude of other areas. Then we had our Dev version which got all the new refactoring. We had written our own code commenting rules and a feature within 4D to make that consistent informative commenting easier. As well all new code - bugs or features, were 'auto' documented in the JERMS system. With these we could decide to move a single new feature into the RDev server as we could track the required code and form changes. As you may have gathered in this description we did not implement all the new shell features into our large main product. We did as we were required to in order of: 1. Required by government participation rules 2. Needed for performance issues 3. Requested by clients and new features of 4D needed to implement 4. A decision to add new feature ideas we had. 5. When the code would be much easier to maintain - simpler. The reason, there was simply not enough time between the various government validation testing to implement the whole new shell. The nice things about writing a new shell is that in the very core security, debugging, communication, and UI features can be made consistent and easier to maintain. It is a good tool to understand our own code, ask ourselves why we do something one way and not another, keeps the code UI up-to-date, takes advantage of performance improvements in 4D, and let's us operate on the newer OSes. Testing is the big hurdle. Writing code to be self testing is expensive in time. It also changes the coding style and changes design considerations. We tried implementing various ways which sounded good but the costs were too much. As well some method of doing this resulted in code that was clumsy because of the self testing feature requirement. Currently we have settled on a standard of various debugging tools we have written to report problems within each method. We can turn these on or off at various levels to expose problems or potential problems. This is not automated testing which is a dream - IMHO - from all the research and trying we have done. We had 8 different governments having requirements and published test scripts that we had to prove the software against. The good thing about this was that we were forced to do a lot of testing of the software ourselves through all of these scripts. Then we had to do them in front of the testers live. This proved a lot of the features and code base that was refactored worked. These tests along with a lot of assert testing, and debug code made it possible to deliver a reasonably robust refactored product. SUMMARY I did this work for 26 years. I learned a lot in that time, and worked far too hard, but the results were pleasantly obvious. We could do the refactoring because of a few things: 1. Large enough installed base to pay for it. Deep pockets are required. It easily cost us $350k a year. I think our product was one of the larger system in the 4D world. Your costs will vary. 2. The owners of the company were not in it to get rich, just pay the bills and provide an exceptional product and services over the long haul. Hopefully get rich at the end. 3. Always looking to do things right (not perfect - but a balanced right). 4. A core set of developers that had been with the product for years, and spent a lot of time with the users. Knowing the reasons why things coded a certain way, broad and detailed knowledge as to features required across multiple jurisdictions, practice styles, and business types. Senior developers that had trained many end users, and still spent a month on the road at user sites. 5. Standard for code style, documenting within and outside the code, JERMS was key for us, as well as the code commentor we built. 6. A CEO that understands the development process and challenges to both insist on the move forward, but understands the time and effort required. 7. If selling the product (I.e. OEM product), a tight reign on marketing people and what they promise. 8. Being a ISO certified company was also valuable. Of course it is not just an automatic improver. The CEO and the main people in the company must take the process serious. A culture within the organization that believes the ISO mantra is required. If not it is too easy to fake it. If you do not have those then I can see why it may not be possible. When it is not possible though there is a price to be paid: 1. Application will be slower 2. Will eventually not run on newer OSes. 3. Interface will look old. 4. If hundreds of new features are required then it will become too hard to maintain. 5. The application will have to be thrown out eventually. Therefore one must decide which pill to take. I think I could write a book on this as I think about the 26 years of the life of that one product. Fortunately, I am in the enviable position of no product, time to write a new shell, then try to get customers, and base the application on his totally rewritten shell based on proven needs, requirements, styles, self debugging, and all the other things I have learned. Sincerely Jody Bevan > On Sep 15, 2016, at 09:07, Benedict, Tom <tom.bened...@optum.com> wrote: > > Kirk Brooks writes:. > >> I just completed a big refactoring of some code... > > I read this kind of statements from time to time and wonder what kind of test > systems are in place which allow 'big refactoring' efforts to succeed. > > As developers we'd love to refactor more, but we've found that, at least for > our app and our business, the testing costs are too great to justify most > refactoring. Even upgrading to a newer version of 4D, which isn't > refactoring, but requires similar test coverage, is a major capital expense > (>$100K not including 4D licenses). > > A few weeks ago Jody Bevan mentioned that they periodically 're-write' the > base shell that their EMR software is built on. I found that astounding. > > So what do folks do to verify that their edits didn't break something? Do you > have automated tests? How do you ensure 100% coverage? And how do you > justify the costs? > > Tom Benedict > Optum > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the intended > recipient or his or her authorized agent, the reader is hereby notified > that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please notify the > sender by replying to this message and delete this e-mail immediately. > ********************************************************************** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ********************************************************************** ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **********************************************************************
