wrote:
I understand that Sharan brought up the framework rewrite subject at
ApacheCon, and some attendees felt that the framework is fine and no action
needs to be taken.
In this message, I will try to give a detailed explanation of why a
framework rewrite is necessary. I don't plan to take any further action on
this subject, because I've brought it up before without success, and I'm
tired of discussing it. It is my hope that the light bulb will click on in
someone's head and they will take action.
My Background
-------------
I became a member of the OFBiz community in 2004. I immediately started
making contributions to the project by supplying patches to the issue
tracker. In 2007, I became a committer. Most of my initial work was on the
UI and some work in the applications (mainly Asset Maintenance and Work
Effort). I stayed away from touching the framework code because it was
deep, dark, and scary.
Eventually, I started to understand how the framework code works and I
made some minor modifications. As my understanding grew, I progressed to
rewriting large swaths of framework code - making it thread-safe, fault
tolerant, efficient, and easier to use.
I will list some of my contributions here, so everyone can have a clear
understanding of my experience with the framework code:
New Features
User Preferences
Visual Themes
Custom UI Label XML File Format
Temporal Expressions
Data Type Conversion Framework
Screen Widget Boundary Comments
Metrics
Integrations
UEL
iCalendar
JSR 223
WebDAV
LDAP
Refactorings/Improvements
FlexibleStringExpander
FlexibleMapExpander
FOP Integration
FreeMarkerWorker
Date-Time Handling
Mini-language
Job Scheduler
In addition, I have performed innumerable framework bug fixes.
So, the contents of this message come from years of experience mucking
about in the framework code.
Okay, let's get started...
Initial Problem Statement
-------------------------
In 2009, David Jones started a framework rewrite in a branch:
https://svn.apache.org/repos/asf/ofbiz/branches/executioncontext20090716
At the time, there was some agreement that a rewrite was necessary, but
there was disagreement as to how the rewrite should be incorporated into
the project:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/200908.mbox/%[email protected]%3E
There were concerns that a rewrite would break backward compatibility.
Work on the rewrite branch stopped. Eventually, Jacopo suggested the
community be more accepting of backward-incompatible changes:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/201004.mbox/%[email protected]%3e
Despite an effort to convince David to proceed with the framework rewrite,
he ended up doing it in a separate project:
http://mail-archives.apache.org/mod_mbox/ofbiz-dev/201104.mbox/%[email protected]%3E
This page describes differences between OFBiz and Moqui, and within it you
can extract information on the problems David was trying to solve:
http://sourceforge.net/p/moqui/discussion/1086127/thread/4c52f240/
There was an email he sent out on the OFBiz dev list where he listed the
problems he saw in the framework, but I can't find it. The rest of this
message will include the issues he mentioned (the ones I remember). I was
in agreement with him at the time, and I still agree that a framework
rewrite is necessary.
The Problems
------------
Code is scattered everywhere - due to an initial effort to make the
framework modular. This causes serious problems. The mere fact that
components like entityext and securityext EXIST makes it clear that there
are problems - those components should not be there. Also, we run into the
recurring problem of circular dependencies (component A will not build
unless component B is built, and component B will not build unless
component A is built).
Bad separation of concerns. There are far too many examples of classes
that try to be everything to everyone. This makes debugging difficult, and
it makes maintenance/improvements a nightmare. [Using an analogy, consider
an automobile design where a spark plug is not separate from a
transmission. Instead, the automobile uses a spark-plug-transmission. So
when the engine is running rough because the spark plug is bad, you have to
replace the spark plug AND the transmission.] A good framework example can
be found in my rewrite of the mini-language code. Originally, the models
AND the script execution context both contained script behaviors - making
debugging/improvements difficult. I changed it so only the models contain
script behavior and the script execution context contains only the script
execution state.
Lack of good OO design. There are many places where a bit of framework
functionality is contained in a single method that is hundreds or thousands
of lines long. There is a term for that: Brittle Code. Code isn't reused.
Instead, it is copy-and-pasted all over - so when a problem is found in the
C&P code, it has to be fixed in many places instead of one.
Fail-slow design. There are a lot of places in low-level code where an
error condition is encountered, but instead of throwing an exception, the
error is ignored and maybe it is logged, or the code tries to "guess" at a
solution and then provide an arbitrary default behavior. I've seen many
developers struggle with debugging a problem because they didn't look at
the logs, or because the error was not logged and there is no way of
knowing what caused it. They end up spending hours single-stepping through
code until it reaches the error.
Out-of-date code. A good example is the use of Javolution. That library
was beneficial in the Java 1.4 days, but it is not necessary today because
of improved garbage collection. Another good example is DCL code. DCL was
used extensively in OFBiz, but it is clearly documented to be an unreliable
design (I can get it to fail 90% of the time). Some DCL code has been
replaced, but a lot of it is still there.
Portions of the API are overly complicated. Some methods require a
collection of user-specified artifacts/arguments, which makes client code
complicated and verbose. (David solved that problem with his Execution
Context.) Portions of the API are cluttered with unnecessary "convenience
methods" - making the API harder to learn and memorize. In some places, a
domain-specific API is spread across instance methods and static methods
and across different classes - making the API hard to understand and use.
Yes, there can be good designs that require something like that, but in the
OFBiz framework, it exists because of a bad design, not a good one.
Use of thread-local variables. This makes multi-threaded design
impossible. The J2EE specification and the Servlet API require one thread
per request (and most J2EE libraries depend on that behavior), so the
current design makes sense from a J2EE perspective, but what if I don't
want to run the framework in a J2EE container? Which leads to the next
problem...
Dependence on J2EE designs/APIs/libraries. There are developers in the
Java community (myself included) who are beginning to question if J2EE is
really necessary to run web applications. The folks at Atomikos are a good
example. OFBiz does not use EJBs, so tying the framework to J2EE does not
make sense. It would be better if the framework was designed to run outside
a J2EE container, and then have container integration as an option.
Configuration files are scattered everywhere. Anyone who has deployed
OFBiz in a production environment will agree this is a problem. Try
changing the HTTP/HTTPS and port settings - it is a nightmare. Some
configuration settings are in nonsensical places.
An abysmal lack of unit testing. I don't have an exact figure for code
coverage, but my gut feeling is coverage is less than 10%. Basically, we
all have our fingers crossed - hoping that the framework code works as
expected. This was made painfully obvious a while back when I was looking
at some entity caching code and thought to myself "this code can't work."
So I wrote some entity cache unit tests and confirmed that the entity cache
had serious problems. Think about that - years passed with no entity cache
unit tests and consequently we had no idea it wasn't working.
Fix Versus Rewrite
------------------
Jira issues could be created for these problems and teams of developers
could work to fix them.
Or, we could create a branch and start over from scratch. This time
around, there should be less push-back from people concerned about
backwards compatibility. A rewrite offers the advantage of reconsidering
everything - like API design, general problem solving, and new features.
I created a Wiki page for a framework design:
https://cwiki.apache.org/confluence/display/OFBADMIN/Another+Framework+Vision
but there hasn't been much interest in it. If the community decides to go
ahead with a rewrite, then please feel free to use the Wiki pages as a
guide.
Sandglass Foundation
--------------------
Like David, I came to the conclusion that a framework rewrite would be
easier outside the OFBiz community. So, I created my own library called
Foundation:
http://www.sandglass-software.com/products/sandglass/documents/Foundation_Brochure.pdf
(PDF)
and I only mention it here to stress how wonderful it can be to start with
a clean slate and design an API that is concise yet powerful. (Please do
not discuss Foundation here, contact me privately if you want more
information.)
Some examples of what can be done with a rewrite:
A single configuration file
Use ANSI/ISO SQL SELECT statement strings instead of constructing
complicated Java structures
Simultaneous asynchronous queries
Relational integrity across multiple datasources
Multi-table SELECT across multiple datasources
Automatic and transparent row version control
Automatic and transparent multi-language datasource support
Abstract entities (similar to SQL user types)
Service engine throttling (protects against server over-utilization)
Simplified security (authorization) (
https://cwiki.apache.org/confluence/display/OFBIZ/OFBiz+Security+Redesign)
Pure interface-based API - so developers are free to modify framework
behavior by using decorators
Thorough unit tests
Benefits of a rewrite:
Reduced resource requirements (lower hosting fees)
Reduce application development time - due to a simplified API
Easier framework code maintenance
Better reliability
Conclusion
----------
Like I said at the start, this is all I will say about the subject. I'm
done trying to convince everyone. I hope someone agrees with me and they
are able to build support for the idea.
--
Adrian Crum
Sandglass Software
www.sandglass-software.com
