Kirk:

I agree that rewriting code, and the associated testing is very expensive for 
large critical system like medical software. You are right about the cost being 
in excess of $100k (easily). With these kinds of cost why would I want to incur 
these costs.  The description is long so you may want to skip the details and 
just go to the executive summary - at the bottom.


1. Old code can be crippling:  Our application started out with version 2.0.10 
(pre server) of 4D. If we did not rewrite code we would have had a very 
crippled multiuser environment, and have old code that did not take advantage 
of the better implementation features of newer versions of 4D. We never could 
have had 120 users on our 4D version 3.x of 4D without a major rewrite. As 
sites got terabytes of data and used the solution for all their work the demand 
on the system performance became greater each year.

2. Many of our competitors in the market sold applications that were very long 
in the tooth. It showed, and it was easy to get new customers interested in our 
products because we refactored our code. Their owners make more money though 
because they do not incur these costs. They will not survive in the long haul.

3. I had the attitude of adding many features every year, and keeping up with 
the UI demands of potential new customers. Government intrusion into the 
business demanded features that were often stupid and never used, but were 
required to 'play' in the market. To be fair they had good ideas too, but they 
got them all from the vendor community that they then passed along to other 
vendors as requirements.  As one of the first players in our market we already 
had them implemented

Let me explain the way we accomplished this.

With an installed base of over 4,000 concurrent users at about 200 sites, and 
add to that the randomness of over 20,000 people using the software one must 
tread very carefully when modifying code. Medical people have little tolerance 
for crippling, or error prone code (I appreciate this as a patient). I know 
there are OEMs out there with larger installed sites, and concurrent users so 
they too know these issues (like Kirk).

With this installed base as well as government intrusion in the marketplace we 
just could not upgrade our solution with each new version of 4D. This was not a 
fault of 4D Inc, but the resources required for testing in multiple 
jurisdictions each year. We had to have people working on validation testing 
almost all the time. This cost is not included in my estimate of the cost of 
refactoring.

With a version of 4D that I decided to rewrite our shell (as I am doing now 
with v15) we start from no code and no tables. We have a list of features we 
want to implement both from the perspective of new features in 4D, new features 
in our code, and better ways to implement something we already have done.

With this in mind the new shell is written. He last time I did this it took me 
4 wonderful months to accomplish. As a programmer those are fun months. We knew 
what the feature set we were writing to ahead of time. Therefore we could write 
it better. We knew where we had clumsy code and could write the features in an 
elegant, obvious, easier to maintain manner. For example when Object Tools came 
out that was a big part of one rewrite, but freeing. Same for Interprocess 
Variables, 4D Server. There are many other over the years. The advantage was we 
could rewrite the code knowing the feature set we required to accommodate that 
the code we wrote 15 or 26 years earlier we never had many of them in mind.

This is the attitude and objective of our rewrite of the base shell. This 
typically includes security (very nice to have in the base shell), interface, 
online help, debugging tools, standard interfaces, administration, user 
handling, network testing, performance testing, automated reporting from the 
client to us, to name just a few.

We would then use this new Shell as to starter to rewrite a small application 
that we used internally to process medical payments of several million dollars 
a week. Sounds scary but with error code detection we knew if it worked well or 
not. It is a small application doing a big job. With this application we got to 
discover some of the bugs and fix them. This typically took about a month of 
developer time.

We then had a piece of software that we used to run our whole business - JERMS. 
This one would take much longer to rewrite - about 6 months. This software was 
used by all our staff. Nice thing is that it was all in house users hammering 
away at the server all day long. In the last 10 years clients accessed this 
system too. The interface was the same was as for our medical application. This 
we worked through with for another six months to resolve issues. We would learn 
a lot about our refactored code and the new version of 4D through this 
experience.

Then the task of refactoring our main product. Before government intrusion into 
the market we could go at our own pace so we could manage this better. Even 
with only one government intrusion we could easily manage the process as there 
was a working relationship between the twenty vendors and the government as to 
the pace of required change. There is a whole different story when many more 
government's got into the demand game.

We would decide what of the features we would refactor in code and of course 
implement the required new features by clients or government. A must of this 
was having two development servers. We called one RDev (Release Dev) which 
would only get bug fixes. This way we could quickly deliver a bug fix without 
worrying about code changes in a multitude of other areas. Then we had our Dev 
version which got all the new refactoring.

We had written our own code commenting rules and a feature within 4D to make 
that consistent informative commenting easier. As well all new code - bugs or 
features, were 'auto' documented in the JERMS system. With these we could 
decide to move a single new feature into the RDev server as we could track the 
required code and form changes.

As you may have gathered in this description we did not implement all the new 
shell features into our large main product. We did as we were required to in 
order of:
1. Required by government participation rules
2. Needed for performance issues
3. Requested by clients and new features of 4D needed to implement
4. A decision to add new feature ideas we had.
5. When the code would be much easier to maintain - simpler.

The reason, there was simply not enough time between the various government 
validation testing to implement the whole new shell. 

The nice things about writing a new shell is that in the very core security, 
debugging, communication, and UI features can be made consistent and easier to 
maintain. It is a good tool to understand our own code, ask ourselves why we do 
something one way and not another, keeps the code UI up-to-date, takes 
advantage of performance improvements in 4D, and let's us operate on the newer 
OSes.

Testing is the big hurdle. Writing code to be self testing is expensive in 
time. It also changes 
the coding style and changes design considerations. We tried implementing 
various ways which sounded good but the costs were too much. As well some 
method of doing this resulted in code that was clumsy because of the self 
testing feature requirement. Currently we have settled on a standard of various 
debugging tools we have written to report problems within each method. We can 
turn these on or off at various levels to expose problems or potential 
problems. This is not automated testing which is a dream - IMHO - from all the 
research and trying we have done.

We had 8 different governments having requirements and published test scripts 
that we had to prove the software against. The good thing about this was that 
we were forced to do a lot of testing of the software ourselves through all of 
these scripts. Then we had to do them in front of the testers live. This proved 
a lot of the features and code base that was refactored worked. These tests 
along with a lot of assert testing, and debug code made it possible to deliver 
a reasonably robust refactored product. 

SUMMARY

I did this work for 26 years. I learned a lot in that time, and worked far too 
hard, but the results were pleasantly obvious. We could do the refactoring 
because of a few things:

1. Large enough installed base to pay for it. Deep pockets are required. It 
easily cost us $350k a year. I think our product was one of the larger system 
in the 4D world. Your costs will vary.

2. The owners of the company were not in it to get rich, just pay the bills and 
provide an exceptional product and services over the long haul. Hopefully get 
rich at the end.

3. Always looking to do things right (not perfect - but a balanced right).

4. A core set of developers that had been with the product for years, and spent 
a lot of time with the users. Knowing the reasons why things coded a certain 
way, broad and detailed knowledge as to features required across multiple 
jurisdictions, practice styles, and business types. Senior developers that had 
trained many end users, and still spent a month on the road at user sites.

5. Standard for code style, documenting within and outside the code, JERMS was 
key for us, as well as the code commentor we built.

6. A CEO that understands the development process and challenges to both insist 
on the move  forward, but understands the time and effort required.

7. If selling the product (I.e. OEM product), a tight reign on marketing people 
and what they promise.

8. Being a ISO certified company was also valuable. Of course it is not just an 
automatic improver. The CEO and the main people in the company must take the 
process serious. A culture within the organization that believes the ISO mantra 
is required. If not it is too easy to fake it.

If you do not have those then I can see why it may not be possible. When it is 
not possible though there is a price to be paid:
1. Application will be slower
2. Will eventually not run on newer OSes.
3. Interface will look old.
4. If hundreds of new features are required then it will become too hard to 
maintain.
5. The application will have to be thrown out eventually.

Therefore one must decide which pill to take.  

I think I could write a book on this as I think about the 26 years of the life 
of that one product.

Fortunately, I am in the enviable position of no product, time to write a new 
shell, then try to get customers, and base the application on his totally 
rewritten shell based on proven needs, requirements, styles, self debugging, 
and all the other things I have learned.





Sincerely
Jody Bevan

> On Sep 15, 2016, at 09:07, Benedict, Tom <tom.bened...@optum.com> wrote:
> 
> Kirk Brooks writes:.
> 
>> I just completed a big refactoring of some code...
> 
> I read this kind of statements from time to time and wonder what kind of test 
> systems are in place which allow 'big refactoring' efforts to succeed.
> 
> As developers we'd love to refactor more, but we've found that, at least for 
> our app and our business, the testing costs are too great to justify most 
> refactoring. Even upgrading to a newer version of 4D, which isn't 
> refactoring, but requires similar test coverage, is a major capital expense 
> (>$100K not including 4D licenses).
> 
> A few weeks ago Jody Bevan mentioned that they periodically 're-write' the 
> base shell that their EMR software is built on. I found that astounding.
> 
> So what do folks do to verify that their edits didn't break something? Do you 
> have automated tests?  How do you ensure 100% coverage? And how do you 
> justify the costs?
> 
> Tom Benedict
> Optum
> 
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**********************************************************************

Reply via email to