James, Thank you for the extremely detailed outline of the benefits and reasons for finishing the move to a mono repo for the Core Grails Projects, by consolidating the remaining two.
It is my belief that 1/3 or more of the time invested over the last 8-9 months was spent on task that are eliminated by a mono repo. Maybe 50%. A mono repo makes releases simple vs taking days which was down from weeks per release. I am in favor of merging Grails-geb & Grails-data-mapping. The details for build time and mongo are well detailed here and I see a clear plan for how they will be addressed with changes before and after these two projects are merged into grails-core. I strongly believe that a mono repo is key to quickly iterating on Grails 7.0.x, 7.1.x and 8.0.x, this year. James Fredley On 2025/04/17 04:05:31 James Daugherty wrote: > Hi Everyone, > > We've previously discussed [1] merging the grails-data-mapping repo into > grails-core. Other than grails-geb, this is the last remaining repository > to merge into core to have a mono repo. This email attempts to summarize > some of the previously expressed concerns and advocates for merging > grails-data-mapping & grails-geb sooner rather than later. Let's gather > people's thoughts so we can determine if a vote thread is feasible. > > Some of the recent concerns that have been raised on merging data mapping > are: > * slower build times (both locally & in GitHub actions) > * the requirement for mongodb > > To answer those concerns: > ------------------------------------------------ > On slower build times: > ------------------------------------------------ > The current grails-core should not be viewed as slow. Across all of the > recent mergers (cache, views, gradle plugins, docs), I've spent a lot of > time optimizing the build. These optimizations include: > A. Converting a substantial amount of our build files to lazy > initialization instead of eager initialization & updating the grails gradle > plugins to make use of lazy where possible. This means we only spend a > total of 8.8 seconds in configuration in a project being built from > scratch. > B. Updating parts of the build & parts of the grails gradle plugins to be > cacheable by defining inputs/outputs. This means there's a higher chance > that if a project dependency doesn't change, it won't rebuild now. > C. Decoupling the gradle plugin dependencies so that there are not circular > references & that its dependencies can be managed separately from > application dependencies (we produce a grails-bom for applications & > grails-gradle-bom for gradle usage now). > D. Eliminating unnecessary steps or processes in the build (i.e. stream > lining the docs workflow) > E. Parallelizing the build where possible (there are known issues that > prevent us from being fully parallel, but we're very close to almost all > projects being able to be run in parallel). > F. Fixing our dependency graphs so that we generate proper platform POMs > and proper gradle modules so dependencies can be calculated correctly (and > quickly). > G. I have added properties to both configure tests that should run on an > opt-in basis and an opt-out basis. This allows selectively running tests > by setting a system property on your build. This allows further focused > development when needed. > > The build for the grails-core library is now approximately 3 minutes if > building from scratch on the most recent Mac hardware (assuming the > libraries are already present locally). The build also peeks at 1.5 gig of > memory usage. It is also highly cacheable - only 30% of the tasks have > remaining cache issues (namely ones related to gsp, gson, and asset > compilation). I believe long term we can get the typical build time down > even lower by improving these processes to be cacheable by gradle, by > further decoupling our build, and by further parallelization. After > merging grails-data-mapping, it should be possible to keep these build > times down. > > Concerning the build times in github, the main slowness is caused by how we > matrix test now with windows, mac, linux across different versions of the > JVM. We also get throttled more when we have to do this across every > repository. Having one repository will mean there's less of a chance of > being throttled. Moreover, we can pursue self hosted build agents to solve > this in the long term. In the short term, if it really becomes an issue, > we can enable gradle caching which will result in very little code having > to run to the aforementioned improvements. > > ------------------------------------------------ > On the requirement for mongodb > ------------------------------------------------ > The requirement for a running mongodb causes several issues: > A. It forces the build to be synchronous for testing mongo related projects > (plugin, mongo core, mongo ext, mongo bson, mongo templates, tck for mongo, > etc) > B. It requires the user to have a running mongo instance. > > We know that B. is fixable by running a mongodb container. More over, if > it could be run for each project, then that also solves the synchronous > execution. We know that running a docker container for B is trivial and > grails development already requires a container runtime to run it's > functional tests with geb. The command for this is: `docker run -d --name > mongo-on-docker -p 27017:27017 mongo` > > A. will then be fixed if we can spin up a container over the lifecycle of a > given project's tests. For example, we could use test containers prior to > the GrailsApp.run() call in Application.groovy to ensure one exists per > application. There will need to be some configuration rework, but it > shouldn't be too hard to accomplish longer term. > > > > ------------------------------------------------ > As for why we should merge these libraries: > ------------------------------------------------ > 1. While working on the merges of the previous builds, I have discovered > numerous validations that gradle performs (circular dependencies, etc) that > were not being performed when these builds are separate. Combined, we get > the benefit of gradle warning us about circular dependencies and benefiting > from this feedback. > > 2. Somewhat related, we can institute code standards, code styles, and code > quality scans in a centralized manner. > > 3. Seperate from Gradle's validations, we can implement our own gradle > plugins local to the grails-core repo that will enforce architecture > separation - this includes ensuring that gsp & gorm can be used separately > from Grails in the long run. > > 4. If a gradle project that is a dependency of grails-core is partially > published, it will break functional tests in all repositories. This means > you have to comment the tests out across repositories until all artifacts > are published again. Inside of the same project, this issue does not exist. > > 3. The iteration time on development is vastly improved in a single > repository. The optimizations I made to the gradle plugin took seconds to > test and I would not have been able to make them in separate projects in > less than a day. The feedback loop is a significant time saver. What > would take 20-30minutes due to build publishing before I was doing in > seconds. The gradle plugin changes would likely have taken over a week (or > longer) if these plugins were still in a separate repository. > > 4. The known issues with grails-data-mapping are not major blockers. While > they may initially slow the build times, we can address the majority of the > time by solving the mongo problem. We have an initial approach that works, > we'll just need to adjust the configuration in the mongo projects to > connect to different containers. > > 5. Having a mono repo ensures that any change to Grails will be tested > fully. Several of us have spent a significant amount of time chasing down > bugs, that we later have discovered are due to someone only running the > tests in the project. If someone changes code related to the core of > grails, they must run all of the associated tests. In a mono project, this > happens locally during development. Outside of it, it happens by users > discovering the bug in a milestone. > > 6. Spending time on the build process - the github action release workflow, > etc - is a significant problem. We don't want to be working on a build > process. We want to be developing code and fixing bugs for grails. > > 7. Spring Boot & Hibernate will have major upgrades on a more regular basis > going forward. To make the changes necessary, we don't want to be working > on separate processes. We need to adopt a more rapid release schedule and > react to library upgrades faster so that we don't end up in the situation > we have been in for Grails 7. The reason Grails 7 development has taken so > incredibly long, is we're updating some libraries that are over 4 years > old. The technical debt can be prevented by updating more often and > staying up to date with upstream libraries - which also ensures the > security of the framework. > > 8. Apache's release process requires a security review. This security > review ensures that builds are not being tampered with and our current > release process across many repos requires build tampering. We eventually > stop modifying a build, but to be able to release a milestone sooner with > the apache coordinates, we need to be in one repository. > > 9. We no longer have admin rights to the GitHub organization we are under. > One of the discoveries we made after moving, was that we can't trigger > workflow actions from one repo to another. Being in one repository, means > we don't have to do that and infrastructure does not need to find work > arounds for our existing processes. > > 10. Contributing to grails will be made easier for newer contributors. > They wont' have to learn to build projects in certain orders or how to work > around issues when something fails. > > > I'm sure I've forgotten several of the reasons, but I'd like to propose we > go to a mono repo for the core libraries that make up a grails release. > Data mapping & geb are the only ones that remain to have this. I'd like to > fast track this and deal with the mongo / slowness after merge. I believe > we can resolve these issues and by merging sooner we can get to releasing > the first milestone under Apache. > > -James > > [1] https://lists.apache.org/thread/sfzzzbb1zo6k4w8hz0ro13wx4n4jyhr6 >
