Hi everyone,
... just wanted to give you a bit of an update on some recent improvements
that have an impact on build speed and development experience:
*OpenJPA plugin*
Petri spent some time getting the OpenJPA plugin to do the enhancement
processing in a way to take full advantage of the build cache. This one is
actually a major speed improvement overall as it allows now for truly
incremental compilation of "fineract-provider" (you'll see, the first
compile takes quite some time... there's a lot of code, but the second one
just skips when no changes to the sources were made.
*Fix Gradle compiler parameters*
This was an odd one. Turns out if you add compiler parameters multiple
times then this slows down the build process significantly (I have no
explanation, just experimentation). That one is on me, I introduced this a
while back, because I didn't see that we can add all parameters at once.
Low hanging fruit.
*Automatic build script version management based on Git tags and Git hashes*
This one is a little bit of a convenience. Until now we manually set (or
not) a property ("releaseVersion") in gradle.properties to determine the
version of Fineract. This is easy to forget when doing the releases. Now
that we have Docker builds based on a Gradle plugin this also comes in
handy when setting the Docker tags (just use the "$version" variable). The
only thing we need to do now are proper releases with Git tags. Git hashes
will be part of the version number during development (pull requests,
develop branch); when we do a release (in master) the version pattern
automatically follows the Semantic Versioning rules (no Git hash). BTW: Git
hashes change on every commit; like this all important artifacts become
uniquely identifiable no matter which branch you are working in.
*Create separate sub-project for WAR build*
We have now a separate module for the WAR build. This was done in
preparation to improve the integration tests, but also to simplify the
"fineract-provider" build script (was needed for the Docker build
improvement).
Note: personally I would drop this deliverable sooner than later; with a
proper Docker image build in place this shouldn't be needed anymore. We
used it here mainly for setting up tests (see my comments below concerning
"Testcontainers"). Is there a significant audience that still uses these
WAR files in production?
*Docker image build with Gradle Jib plugin*
I'm pretty sure up until now the Docker image build via the
docker-compose.yml file was slowing down pretty much everyone. On my
machine this took > 10min, because the Dockerfile contained one section
that did a Gradle build inside a Docker container and the results of that
build were then used to populate the actual Docker image. Because
everything ran in a Docker container that build could not take advantage of
any caches outside of it where we did pretty much the same. So, instead of
having to build twice we now only have one run that takes advantage of
Gradle's build cache and the Docker image is built now by Google's Jib
plugin which is very fast (just a couple of secs) and does some
optimizations to keep the images small.
*Replace Tomcat Cargo plugin to run integration tests with Testcontainers*
This is a major improvement for the development experience. Until now it
was a bit complicated to run the integration tests locally. You needed a
separate MySQL instance running that required some initialization (see
Gradle script "createDb" tasks). This also made it a bit inconvenient to
run those tests in your IDE. Not anymore! Now that we have a Docker image
as part of the build process we re-use it immediately for those integration
tests. The Testcontainers framework brings Docker to JUnit 5 tests with
not much effort. Actually, the only changes I applied to the existing tests
were to make sure that the proper ports are used (Testcontainers uses
random ports which allows you to run - in theory - multiple Fineract
instances side by side) and in some cases I needed to enforce a certain
execution order of test functions, because they had concurrency issues.
Looks like not all tests are thread safe. Which reminds me that I've
tentatively enabled parallel test execution to speed up things.
I also got rid of the separate modules for OAuth2 and 2FA integration
testing; this can now be done in the same module. Simpler Github Actions
workflow.
Note: this PR is still in progress, but looks already promising. More
changes on the testing front coming (Cucumber, faster JUnit tests with
minimal or no Docker container dependencies)
*Simplify selection of auth strategies*
As you know we decided on build time which authentication scheme to use.
Additionally you had to use certain command line parameters to activate
related Spring profiles. This was a bit redundant, especially since we had
4 different "application.properties" files that we copied into the
"fineract-provider/src/main/resources" folder. This was one of the main
reasons why we constantly did a full rebuild; "src/main/resources" is
monitored by Gradle for changes... and obviously if we copy a file over and
over again into that folder (which changes its timestamp even if there are
no changes) then Gradle assumes we have a change and triggers the build.
There is still some room for improvement, but it behaves already a lot
nicer now.
*Upgrade to JDK 17*
JDK 17 is already out for a while and it's an LTS release, so we upgraded
everything (Gradle build script, Docker image, Fineract) to be compatible
with it. Changes were mostly command line parameters to allow access to
certain JDK modules. There was one interesting change necessary concerning
"GsonBuilder"; we were instantiating that class in quite a lot of places
(mostly those manual mapper classes that we use to map JSON to Java
classes). The problem with just calling "new GsonBuilder()" was that this
doesn't automatically register all necessary Gson serializers (in this case
JDK 17 complained that the serializer for "java.time.LocalTime" was
missing). We have now a static function
"GoogleGsonSerializerHelper.createGsonBuilder()" that ensures that all
these serializers are properly registered.
There was one other rather curious issue related to date formats. In one of
the tests we had a date string "15 Sep 2015" (and others like "15 March
2015" etc.); but this particular one for September always failed in JDK 17
with the date pattern "dd MMMM yyyy" applied. This September date is not
accepted anymore in JDK 17. The correct value is now "15 September 2015".
This kind of makes sense as "Sep" is an abbreviation and the other months
are formatted/parsed with full month names; someone must have cleaned up
this behavior in the latest JDK release.
*Coming soon...*
- Introduce JReleaser: JReleaser includes a lot of best practices for
creating releases; I'll replace my home grown script (and the somewhat
lengthy description on how to run it) with this plugin; should make life a
lot easier.
- More Github Actions Docker build cache improvements: I'll try to
preserve some of the already downloaded Docker layers to improve the
overall build speed
- More build speed improvements: at the moment the fineract-client
module is still triggering a full rebuild too often; similar situation with
the Docker image build plugin. Both tasks don't take that long, but still
would be a nice tweak.
- Removing implicit dependencies: we have quite some places where we
explicitly force the execution of tasks to make sure that everything is
executed in the proper order; this doesn't really work very well to get the
most out of caching. Removing these dependencies should speed up builds
quite a lot.
I think that's it more or less for this round. Thanks again to @Petri
Tuomola <[email protected]> for reviewing all these changes and
contributing the OpenJPA plugin improvements.
README contains more detailed information on some of these changes. Please
let me know if something is missing.
FYI
Cheers,
Aleks
P.S.: ... in the meanwhile I did a full integration test run locally on my
machine, just to quickly check where we are. Parallel execution would take
probably only half the time (around 30-40min), but at the moment there are
just too many concurrency issues so reverted back to sequential test
execution. The current execution time on my machine for all integration
tests is > 2h with 530 tests executed of which 27 failed (still running...
"ClientLoanIntegrationTest" is quite a monster). Not too bad for a first
shot; currently trying to make it faster while trying to keep the tests as
they are.
