Hello, What about adding observability to the building process? In this way we can have information about the root causes of the issues during the building time.
A OSS suggestion: https://github.com/craigatk/opentelemetry-gradle-plugin Regards Victor El dom, 1 sept 2024 a las 10:06, James Dailey (<jamespdai...@gmail.com>) escribió: > +1 agree > How do we tackle this ? > > On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <a...@fynarfin.io> wrote: > >> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a >>> horror to find out which ones failed, and our best bet is to text-search >>> for "FAILED" in the 17000+ lines of output. >> >> +1 >> >> Items like this should be high priority in CI improvements epic. The >> overall failed test history, flaky test history, log tracing problem >> (actual result is often not sufficient to quickly debug and linking logs to >> a failed test case is important) were historically connected to getting PRs >> merged in time. CircleCI has helped us with the first two parts of this >> problem. >> >> just by splitting up the code as is and moving some classes to different >>> project modules is not really taking into account the interdependencies... >>> if we don't address those then the project modules are not really helpful >>> (actually it's adds more work to the Gradle builds) >> >> I had overestimated what Fineract modularization design efforts have >> achieved. Would be interesting to know more about the design here as >> "splitting up the code as is" can be helped with more clarity. >> >> >> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic < >> chee...@monkeysintown.com> wrote: >> >>> Kristof, >>> >>> ... great overview, so first: thanks for that. Some notes... >>> >>> Repeatable builds: we should consider this as a major bug and not just >>> "gloss over" it by executing the build multiple times... stuff like this >>> should halt all current development on other features and everyone (or at >>> least one person) should focus on fixing this as soon as possible. I would >>> consider this also a show stopper for a release. >>> >>> Dirty checks: we were actually at one point (2+ years?) not too bad >>> here, but it seems that this is now actually worse than before... my bet is >>> that we introduced some performance hogs by splitting up the code base into >>> separate project modules; this didn't really fix the main problem of the >>> source code (too many interdependencies between packages aka "modules"); >>> just by splitting up the code as is and moving some classes to different >>> project modules is not really taking into account the interdependencies... >>> if we don't address those then the project modules are not really helpful >>> (actually it's adds more work to the Gradle builds), neither for publishing >>> them on Maven central nor chopping up the code into smaller pieces and >>> taking full advantage of caching. >>> >>> Caching: disabling caching is just not such a good idea... at all... >>> again: I would consider this a major bug and halt all machines until it's >>> fixed; also: consider this a show stopper for releases. >>> >>> Console readability: we've talked about this already on occasions (there >>> might be even a mailing list message)... on every upgrade of the >>> dependencies (including the quality assurance tools like Errorprone etc.) >>> the screws get tightened (read: rules get stricter); I remember that after >>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule >>> enforcements, because the reported style and quality checks literally >>> exploded; holding back on Errorprone (and similar) upgrades is obviously >>> not a solution; I tried to be conservative (just disabled those that >>> brought the build to a halt aka that were reported as errors), but as we >>> can see there are enough warnings left to drive everyone crazy; btw: >>> disabling the rules with certain annotations is neither a solution, that's >>> also avoiding to fix the quality issues. >>> >>> Suggestions: >>> >>> - let's make an effort to reduce the interdependencies between >>> packages aka "modules" before we rip the code apart and move it to >>> separate >>> project modules; if we can't do that (backwards compatibility, not >>> allowed >>> to "break eggs" etc.) then let's leave the code in one place as is... I >>> see >>> no added value of having classes in different jar files when it hurts >>> build >>> performance >>> - let's spend time and fix those style and quality issues, not just >>> pile feature on feature on feature... I am pretty sure that everyone can >>> grab a handful of those warnings and fix them as part of their PRs (even >>> if >>> it's not one that is explicitly dedicated to quality assurance), in other >>> words: everyone look a bit left and right to see what ELSE you can >>> fix/add >>> in addition to your tasks (chores vs cherry picking)... fixing everything >>> in one PR is obviously not practical; we'll have a thousand files >>> changed... this will interrupt everyone else quite significantly and is a >>> nightmare to review >>> - there might be less than a handful of Gradle build constraints >>> that are again too paranoid and mess up the build; I think it's worth >>> looking at the OpenAPI yaml build... that one had issues in the past with >>> proper dirty checks, but there might be other/new constraints that are >>> equally unhelpful >>> >>> My 2 cents >>> >>> Aleks >>> >>> >>> >>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kristof.jo...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> After working with Fineract for some months now, I collected several >>>> issues related to the Gradle build system we have in place. I categorized >>>> my pain points, and would like to initiate a discussion on the topic with >>>> the community. My usage scenarios include building locally with >>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github >>>> build from the web UI. >>>> >>>> The problems I gathered fall into the following categories: >>>> >>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails >>>> for the first time with: >>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind >>>> FineractProperties to the 'fineract' property >>>> and simply trying it again always makes it wor0k. >>>> >>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline character >>>> into the Loan class) and restarting Fineract makes it "build" for 17-18 >>>> seconds (checking all modules to be up-to-date) before starting it again. >>>> This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating up to >>>> 800% CPU and spinning up all the vents in my rig while doing so. >>>> >>>> - *INTEGRATION TESTS*: the same dirty check problem happens when >>>> working with tests as well. Adding a blank line to an existing test class >>>> and re-running the test again happens after a ~20 seconds "rebuild" >>>> process. This leads to painfully slow code-test cycles at local >>>> development. >>>> >>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the Gradle >>>> daemon repeatedly grows up to 8Gb memory usage. That causes issues with dev >>>> boxes having 16Gb or even less RAM.. Even using my high-spec dev box, I >>>> needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12 >>>> -Dorg.gradle.priority=low" to keep my desktop responsible during local >>>> builds. >>>> >>>> - *CACHING*: I see many team members and mailing list members >>>> recommending `--no-build-cache --no-daemon` settings when invoking >>>> gradle, to avoid issues with our build. This kind of denies even any >>>> potential adventages of using Gradle over more stable build tools. Note, >>>> that I do not use these settings (so my issues are definetely not caused by >>>> this), but I keep getting this recommendation for a reason. >>>> >>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting >>>> running gradle with `--console=plain` to get some chance to read the >>>> logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+ on a >>>> Github build, it's a horror to find out which ones failed, and our best bet >>>> is to text-search for "FAILED" in the 17000+ lines of output. >>>> >>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check >>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682 >>>> for a perfect example. Here at line 1489 Gradle shows >>>> > Task :fineract-loan:compileJava FAILED >>>> and shows a couple of warnings below, but the actual error is shown at >>>> line 1393 which appears to be part of >>>> > Task :fineract-document:modernizer >>>> which is clearly misleading, and your best bet to find the actual error >>>> on Github is to search for "error: ", but NOT using Ctrl+F, but the >>>> search box at the top right corner. That's ridiculous - a sensible build >>>> tool should communicate build errors in a straightforward way. >>>> >>>> I'd like to ask you to share your experiences and pain points with the >>>> current build that we have and use every day. Once we have a clearer >>>> understanding of the areas that need improvement, we can discuss potential >>>> strategies to revise our build system and ensure it meets basic >>>> requirements.To be fair, please also make sure you share any recent >>>> positive experiences related to Fineract builds, if you have any. >>>> >>>> Thanks, >>>> Kristof >>>> >>>>
