Re: Proposal to modify the testing framework for OFBiz

Ron Wheeler Tue, 19 Jul 2016 19:38:22 -0700

I think that your point was well received.
I am just happy that testing is getting attention.


Ron

On 19/07/2016 7:53 PM, Scott Gray wrote:

I'm not looking for any sort of response really.  I was just speaking in
favor of integration tests because I think they provide better coverage at
the cost of a little bit of speed (the framework has to be running to run
them).  Most of the responses coming in were of the form "unit tests are
great because [insert a reason that applies to any type of test]".  So I
was merely pointing out that the arguments weren't specific to unit tests
and hence weren't really adding value to the discussion.

Regards
Scott

On 20 July 2016 at 02:05, Ron Wheeler <[email protected]>
wrote:

It is not clear what you are expecting as a specific response.

As you move up the food chain, the definition of "units" changes and the
tests get more complicated and the design of the tests gets more complex
(mock objects, testing race conditions).

It is hard to make specific comments about what kind of testing will give
the highest level of confidence at each level but we are going in the right
direction.

The fact that we are considering tests as a major development activity is
a big step forward.

We can argue about specific cases (scope, tools, value, etc) as we get to
them.

I think that we are at the "good idea" stage rather than at the "policy"
stage at this point.

Ron



On 19/07/2016 4:23 AM, Scott Gray wrote:

I think people are missing my point because they keep replying with
generic
statements that aren't specific to unit tests.

On 19 July 2016 at 20:08, Montalbano Florian <
[email protected]>
wrote:

Hi everyone,

unit tests and integration tests are complementary and with test, the
more
the better.
Implementing unit tests will be a great load of work but it is the same
with every kind of test. As Taher said, we will gain a lot from those
unit
tests for avoiding regression while refactoring the framework. From the
start component scope, it has already be proved useful.
And what Ron said about encouraging the creation of "smaller class that
can be tested easily" is very important too. It can be a first step to
reach TDD in the OFBiz development. TDD can help to give confidence to
newcomer and veteran when updating some part of the code and as Hans
said,
this lead to "continuous improvement and finally continuous deployment".

I'm aware that this does not solve the technical implementation stuff,
but
I think this is an improvement that need to be supported.

Have a nice day,

Florian Montalbano


Le 19/07/2016 09:04, Pierre Smits a écrit :

As with anything, the law of diminishing returns also applies to OFBiz

and
tests. This is not true for unit tests and system integration tests, but
also for user acceptance and performance tests.

Nevertheless, the work done up to now is a good start and - I feel
confident - appreciated. And unit tests are certainly valuable in the
framework stack. How it will be for functions (regarding components in
application and special purpose stack) needs to be addressed when we
reach
that bridge.


Best regards,

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Tue, Jul 19, 2016 at 8:22 AM, Taher Alkhateeb <
[email protected]

wrote:

Hi Scott,

Well spoken thank you. Okay may I suggest that for any such work we
will
discuss it here see its Merit and if it makes sense then we take it in.
It
is a little early to discuss it right now because we did not yet go to
the
higher-level components. Once we do I'll be sure to have a conversation
about this in here and would appreciate your input to it.

Regards,

Taher Alkhateeb

On Jul 19, 2016 9:00 AM, "Scott Gray" <[email protected]>
wrote:

Yeah I'm sure unit tests probably worked well for the start component
given

it is the lowest level component in the system and closest to a basic

java

app.  I just think the value proposition might decrease the further up

the

stack we move with them.  I'm not against unit tests when they prove

useful, but further up the stack I think we should prove the case for

them

before doing much work to support the mocking that will be required to

keep

them inline with 'best practices'.

In OFBiz "bad or unwanted" tends to come mostly in the way of
increasing
complexity from features that require more effort to maintain than the
value they provide to the community.  I think there's a chance
attempting
to mock service tests could fall into that bucket.  I could be wrong,

but I

think it's worth looking into before we declare that unit tests are the

best form of testing for OFBiz.

Regards
Scott

On 19 July 2016 at 17:37, Taher Alkhateeb <[email protected]
wrote:

Hi Scott,

Thank you for the feedback. To be focused exactly on integration vs

unit, I

already mentioned above that at least for me the main objective is to

confidently and quickly run the tests in short bursts of red-green
refactor. This allows me to refactor code without waiting in front of

the

screen in between test cycles thus giving me immediate feedback on any

errors I made. Perhaps my intro was too long so this is the squeezed

out

version of it.

I already had one round of testing in the start component which was

much

faster that way and had an immediate impact. Oh and by the way, you
cannot

test the start component with integration tests for example unless you

do

it from an external component which cannot access package protected
items.

This style of coding is applicable I think to any software project

inclusive of OFBiz. Maybe in certain components more than others or

certain

areas more than others. But I can't see how it could be bad or

unwanted.

Taher Alkhateeb

On Jul 19, 2016 8:18 AM, "Scott Gray" <[email protected]>
wrote:

Thanks Ron and Taher for your responses, I appreciate them but I
don't

hear

much in the conversation that speaks to the value of unit tests over

integration tests.  Most of the thoughts shared speaks more to the

value

of

tests rather than differences in the type of tests.

Speed: At an application level we have ~685 tests that run in 35

seconds

(excluding build and data load).  Another point is that there isn't
much
reason why tests can't be run in parallel rather than sequentially.

TDD: Integration tests in OFBiz are as simple if not simpler to
write

than

unit tests and just as useful for TDD.

I'm not sure if I missed any other points raised regarding

integration

vs.

unit tests?

I'm not looking to start a big long debate on the topic, I just

wanted

to

speak out that someone out there (me) doesn't think unit tests are

the

best

solution for testing OFBiz applications.

Regards
Scott

On 19 July 2016 at 16:52, Taher Alkhateeb <

[email protected]>

wrote:

Hi Scott,

Thank you for your input. Your ideas are thought provoking and

enriching

to

the conversation.

The way I look at it, the general rule is usually many unit tests,

less

integration tests, lesser functional tests. So we are not excluding

any

types of test, all of them are important in certain areas for

certain

purposes with certain quantity. Usually integration tests are less
because

as you said they just grab more. I like the picture below as a

general

guideline

http://i.stack.imgur.com/fjQvQ.png

As you already mentioned unit tests are useful for the framework. I
discovered errors in code that I wrote which I was very very

careful

with.
I immediately learned the lesson that humans are not designed to

code,

and

TDD gives you confidence as you build your code with those short

30-60

second red-green refactors. I feel much much safer and more

confident

writing code that way, the test also documents how to use the api,
refactoring and feature change becomes less terrifying. Also messy

code

is

usually not test friendly thus refactoring and unit tests go

hand-in-hand

for improving the code base.

Also, I am sorry to say that the framework code is rather messy and

brittle

in many places. I think you probably encountered this yourself. The

same

is

said for the applications. Now If we start refactoring without unit

tests

then we are back to scary business. So much can go wrong so fast
and

break
things you never expected to break. The framework with all

applications

require heavy refactoring of things like massive ugly methods,

sandwiched
logic, heavy shared mutable state, hidden dependencies, poor

interfaces, and much more. Every time I touch something I get

shocked

at
how bad it looks, spaghetti logic to no end. If you refactor with

integration tests instead of unit tests then you will come down to

a

screeching halt as you wait between these test cycles. My computer
actually

takes 10 minutes or more for a full clean load data and testing.

Now talking about mocking you raise some interesting points, what

can

you
mock vs utilizing integration tests? This is an excellent question

that

does not have a simple answer, and you mostly learn as you code,

difficult

to envision without coding. However simple things like Java

services

can
be

mocked with a standard mocking class that you use everywhere.

Transactions

are difficult to mock and better left to integration tests I think.

SECAs

might be mocked by simply passing output to input as chaining
methods.

Mind

you I am not 100% sure of all of this, coding is the ultimate guide

to

what

can or cannot be done.

So I am not suggesting unit tests as a best-practice (even though

it

is).
Instead I suggest it as something that I and others did and got a

huge

psychological relief and confidence and comfort from. Swimming in

code

without tests is a terrifying business, made more terrifying if the

code
is

bad. Short bursts of red/green/red/green makes you feel good as you

build

up your logic. And I don't have an exact vision of how to do it,
because
the details always win.

I look forward to hearing your thoughts and thank you for enriching

this

conversation.

Taher Alkhateeb

On Tuesday, 19 July 2016, Scott Gray <[email protected]

wrote:

I know I'm late to the party here, but I just want to say that I

think

integration tests have far greater value to OFBiz than unit

tests.

Mostly

because we tend to have quite a low number of tests and

integration

tests

give us much better coverage per line of test code and the tests

are

much
closer to the real world scenarios the application might

encounter.

I don't really see how unit tests could be applied to

non-framework

testing

in a useful manner, could you expand on your vision in that

regard?

mean

would we be testing something smaller than a service as the

'unit'?

What

would we mock? Would transaction management still be active? What

happens
when the service calls another service, I guess we mock the
response

from
that service (how)?

It just seems a very complicated method to achieve a less

thorough

but

albeit faster (maybe) test result.

A build, data load and full test run takes 4m 9s on my laptop.

Excluding

build, data load and framework tests: the application level tests
take

35s,

not very expensive IMO.  The data load time can be reduced to

practically

nothing by copying a clean slate database into runtime for each
run.

I'm mostly just suggesting we be wary of adding complicated
testing
procedures in the hope of achieving some 'best practice' result

which

in
reality will provide minimal benefits.

Regards
Scott

On 18 July 2016 at 18:57, Taher Alkhateeb <

[email protected]

<javascript:;>>

wrote:

Hello Akash,

Fantastic, I have a few unit tests almost done to be included

in

the

start

component. I will create a new subtask under OFBIZ-1463 to

commit

the

tests

so you can use them as a reference if you like to.

I also recommend that you follow the same directory structure

between

the
test code and production code. So for example:

Production code:

framework/start/src/main/java/org/apache/ofbiz/base/start

Test code:

framework/start/src/test/java/org/apache/ofbiz/base/start

The benefit of this hierarchy is that you can access non-public
(package

protected) methods for testing. This is in fact exactly what I
needed

to
be

able to apply some of the tests.

Cheers,

Taher Alkhateeb

On Mon, Jul 18, 2016 at 9:22 AM, Akash Jain <

[email protected] <javascript:;>>

wrote:

Thanks Taher for nice initiative!

We are planning to written unit tests to all components under

OFBIZ-1463

Thanks and Regards

--
Akash Jain

On Mon, Jul 18, 2016 at 10:36 AM, Taher Alkhateeb <
[email protected] <javascript:;>> wrote:

Hello Everyone,

In reference to this thread and the Jira OFBIZ-7254, I'm

very

happy

to

announce that OFBiz is now ready for applying unit tests

with

only

new

lines of code in the build script (r1753143) :)

I recommend we do the following moving forward:

1- Introduce unit tests as much as we can to all components
2- Migrate most of the integration tests we currently have

to

unit

tests

(since they are designed to do very little integration).

This is a great chance for us to practice real TDD in OFBiz

Cheers,

Taher Alkhateeb

On Mon, Jun 13, 2016 at 7:43 AM, Pranay Pandey <
[email protected] <javascript:;>> wrote:

+1

Best regards,

Pranay Pandey
HotWax Systems
http://www.hotwaxsystems.com/

On Fri, Jun 10, 2016 at 7:46 PM, Taher Alkhateeb <
[email protected] <javascript:;>

wrote:

Hello Everyone,

I was able to get a few tests running and this is very

doable.

But

faced

a big problem in designing the testing framework

because

of

ANT.

The problem

----------------

The way the build scripts are designed in OFBiz are

very

complex. A

master

file calls other files which call other files. And in

the

middle

you

have

external libraries (ant-contrib) and macros, and

variables,

and

class

path

declarations, and and and ....

I cannot declare the tests programmatically (with JUnit

test

suites)

because this means lower level components would depend

on
higher

level

components. So I have to do it in ANT, by navigating

this

maze

of

build

scripts, and it was a headache for me just to read

them,

let

alone

modify

them to create a testing framework.

Suggested Solution
------------------------
I suggest to implement the testing framework in Gradle,

and

simply

call

it

from within ant. This is a middle solution that

sustains

ant

for

now,

but

can allow us to switch out later.

This means I will just add one more file called

build.gradle

in

the

top

level directory, and figure out the business logic for

calling

the

test

suites from that file

I look forward to your feedback.

Regards,

Taher Alkhateeb

On Wed, Jun 8, 2016 at 6:00 PM, Taher Alkhateeb <
[email protected] <javascript:;>>
wrote:

Hi Everyone,

Thank you all for your support, JIRA created in
https://issues.apache.org/jira/browse/OFBIZ-7254

I will start working on it and try to implement ASAP

to

get

my

focus

back

on refactoring.

Cheers!

Taher Alkhateeb

On Wed, Jun 8, 2016 at 4:58 PM, Deepak Dixit <
[email protected] <javascript:;>>

wrote:

+1

Thanks & Regards

--
Deepak Dixit
www.hotwaxsystems.com

On Wed, Jun 8, 2016 at 7:12 PM, Mridul Pathak <
[email protected] <javascript:;>>

wrote:

+1

Makes perfect sense.

--
Thanks & Regards,
Mridul Pathak
Senior Manager
HotWax Systems
http://www.hotwaxsystems.com

On Jun 8, 2016, at 2:41 PM, Taher Alkhateeb <
[email protected] <javascript:;>>
wrote:

Hello Everyone,

After refactoring the start component and while

starting

on

the

base

component I realized that the testing framework

for

OFBiz

is

not

good.

You

cannot do real test driven development or

red-green-refactor

with

the

current setup, hence my proposal to change it. I

explain
below:

Problem with current design

----------------------------------------

- What we have right now is not unit tests, it's

really

integration

tests.

You have to start the framework, the database,

the

service

engine,

the

entity engine and pretty much everything.

- Testing is very slow, because it's an

integration

test

as

mentioned

above. 10 minutes on a good computer!

- There is zero mocking! We actually have to

--load-data

for

things

to

work. Again, these are integration tests.

- Too complex: Integration tests by their nature

are

grabbing

too

much.

Mind you, I am not objecting to integration

tests

(I

actually

like

them)

but I am objecting to not having real

unit-tests.

Unit

tests

should

all

run

in a few seconds.

Proposed solution
--------------------------
- We keep what is considered real integration

tests

the

way

they

are

right

now and keep using them

- We move what should be unit tests into simple

JUnit

classes,

and

we

do

not run them using java -jar ofbiz.jar --test,

but

instead

run

them

directly from the build.xml script, so these

files

are

not

identified

in

any XML document, but are simply called

immediately

from

the

build

scripts.

- We clearly mark the difference between

integration

tests

and

unit

tests

(inside the source files or in the suite

declarations).

- We change the run-tests target in build.xml to

run

both

unit

tests

and

integration tests.

I intend to heavily refactor the framework and I

would

feel

better

about

introducing this change while refactoring. What

do

you

guys

think?

Ideas?

Suggestions? Approvals and thumbs up?

Regards,

Taher Alkhateeb

--
Ron Wheeler
President
Artifact Software Inc
email: [email protected]
skype: ronaldmwheeler
phone: 866-970-2435, ext 102



--
Ron Wheeler
President
Artifact Software Inc
email: [email protected]
skype: ronaldmwheeler
phone: 866-970-2435, ext 102

Re: Proposal to modify the testing framework for OFBiz

Reply via email to