... not sure if I understand the end of your message: "consider when doing implementation"...?
But to answer your question: - yes, I'd like to vote on this.... because without this change (or something similar... happy to drop this if someone proposes something better) there will be no type safety and we will live with the manual JSON parsing boilerplate code probably forever - concerning discussion: I hope we will continue to discuss this, because this will be like an ongoing effort... there are approx. 50 "modules" that need to be converted to take full advantage of this change... that obviously won't happen overnight... and given the diversity of implementations I am pretty sure we need to adapt a thing or two down the road; for now I think this approach looks good... unless someone sees an issue that I'm not aware of - as mentioned earlier: no critical services will be touched when the first "module(s)" will be migrated... and once we have them we should pause a bit and do some performance testing - have to find some time to wrap this up, but had to introduce command persistence, because Manoj pointed out that maker-checker feature will also be needed; I want to keep command processing as focused as possible, but persistence is unavoidable and will be needed - maker-checker will not be included in this PR, because it's a different concern/domain (however you want to call it); this can be introduced separately... which helps I guess to get this PR out of the door when things don't get too convoluted The only blocker at the moment - other than the final vote - is a build error that I have to fix. Hope that clarifies things... if not please let me know... On Thu, Mar 13, 2025 at 7:48 PM James Dailey <jdai...@apache.org> wrote: > Alexander - Do you want to continue this discussion and get to a vote? > > > https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5%3A+New+command+processing+infrastructure > > > Really appreciate the conversation thus far. > > It seems it faces no opposition (clarification yes), but I don't know if > there's a real appetite to fully implement across the entire code base... > so I guess that's my concern, that once started it gets completed. Not a > reason to drop it but something to consider when doing implementation. > > Thanks, > > > > On Mon, Mar 10, 2025 at 6:28 AM Aleksandar Vidakovic < > chee...@monkeysintown.com> wrote: > >> Hi Terence, >> >> ... first: thanks for your encouraging and insightful comments. >> >> The current status of this proposal: >> >> - I've added the capability to save incoming write requests (aka >> commands) to the database (brings us a bit closer to the initial idea of >> CQRS replayable commands to restore the system at any given point in >> time); >> there is some error during the build process that I still have to track >> down >> - Manoj made some comments around maker-checker functionality in the >> current production version of Fineract. I didn't reply extensively to >> that, >> but my view is that command processing and maker-checker are two different >> problem domains that should not be mixed together. I'd like to address the >> maker-checker concern in a different proposal that would (maybe) help us >> on >> multiple fronts: >> - maker-checker is more related to human tasks/interactions in a >> workflow... maybe (!) we can address this in a way that we can later >> optionally (!) drop in a proper workflow engine... >> - having maker-checker as a separate component should have the >> same benefits as command processing (type safety, clear readable >> execution >> path, extensible/replaceable, better documentation) >> - maker-checker is tightly related to our security model; an >> improved implementation should help also with a long standing effort to >> modularize the security of Fineract; we have a classic RBAC model baked >> into Fineract that can't really be replaced without major refactoring; >> even >> adding roles/permissions to adjust to the deployment environment is not >> really easy... let alone using some more advanced concepts like Open >> Policy >> Agent or OpenFGA >> - again, this would be subject to a separate proposal... don't >> want to overload >> >> The first services that will be migrated to the new command processing >> are free of maker-checker requirements and are NOT critical ones (like >> loan, general ledger etc.). I think that way we won't create a monster pull >> request and validate the two improved concepts (command processing, >> maker-checker) step by step. >> >> Cheers, >> >> Aleks >> >> On Mon, Mar 10, 2025 at 11:14 AM Terence Monteiro < >> terence.plac...@gmail.com> wrote: >> >>> Hi Aleks and fellow Fineract Devs, >>> >>> Just wanted to bump this thread and share some thoughts.. >>> 1. I'm certain of the benefit such accurate typing and class usage for >>> data interchange will bring. From my experience with both strongly typed >>> (C++, Java, Python) and weakly typed languages (C, Perl5, PHP) the >>> flexibility of the latter and not using classes becomes a challenge when >>> the project scales up to around 10000+ lines of code. I think Fineract is >>> at the stage of its evolution where this kind of work will bring a sea >>> change in ease of ongoing Development and auto-generated Documentation. >>> 2. I like the bullet point summary of changes expressed in the >>> fineract-command module README from the PR. From an architecture >>> standpoint, I see a far reaching impact such a change can have. I feel the >>> questions raised after the initial mail by Aleks have been satisfactorily >>> answered so I'd like to know whether we have progressed beyond the PR >>> shared in the initial mail. >>> 3. Practically having worked with developers of varying degrees of >>> experience and IDE (Eclipse, Intellij-Idea, VS Code) adoption and >>> utilization, this will yield, if done properly, much more accurate >>> auto-generated Class names for function signatures and accurate >>> implementation stubs. Circling back to the earlier point about the stage we >>> are at with Fineract having all these lines of code, I feel this FSIP can >>> lay the groundwork for a lot of contributions to >>> a). Clean, refactor current code >>> b). introduce new features >>> c). Inject more quality API consuming projects like Web app and >>> mobile app from Mifos, as well as the dozens of open-source and >>> proprietary consumer apps >>> >>> Given this promise and opening of possibilities, I strongly support this >>> proposal. I'd be glad to assist in any way possible working you Aleks on >>> fleshing out required parts of making this happen as long as I can get some >>> more native Java eyes to review (I think in algorithms preferably through >>> an OOP lens, but don't always think in Java unless deep into a Java >>> project). >>> >>> Regards, >>> Terence Monteiro. >>> >>> >>> On Fri, Jan 31, 2025 at 12:21 AM Aleksandar Vidakovic < >>> chee...@monkeysintown.com> wrote: >>> >>>> ... thanks Manoj for reviewing the proposal. The additional features >>>> you mentioned can be of course added... as soon as I have a moment I'll >>>> have a stab at this and will provide an example in a unit test that >>>> demonstrates it. >>>> >>>> Cheers >>>> >>>> On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <mano...@apache.org> >>>> wrote: >>>> >>>>> >>>>> Hi Aleks, >>>>> >>>>> Thank you for your initiative in proposing this system enhancement. I >>>>> strongly support this change, as it addresses a critical need in our >>>>> current architecture. >>>>> >>>>> The existing synchronous command handler facilitates the maker-checker >>>>> workflow. It allowed direct storage of serialized commands (including >>>>> request payload JSON) in the database, enabling easy retrieval and replay >>>>> during checker approval. As we transition to the new approach, Are there >>>>> proposed modifications to the maker-checker workflow itself? >>>>> >>>>> Additionally, the existing implementation includes a permission check >>>>> (authorization) prior to invoking the handler, as well as a Hook Event >>>>> Processor integrated into the control flow to invoke external APIs >>>>> asynchronously. To ensure continuity of these functionalities, can we >>>>> include these in the proposed command executors? >>>>> >>>>> This will help maintain critical auditability standards while >>>>> adopting the updated architecture. >>>>> >>>>> Regards, >>>>> Manoj >>>>> >>>>> On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote: >>>>> > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>>> > victor.rom...@fintecheando.mx> wrote: >>>>> > >>>>> > > Hello, >>>>> > > >>>>> > > I have been reading the discussion, the FISP and the GitHub PR... >>>>> this >>>>> > > must receive more feedback from the community. >>>>> > > >>>>> > >>>>> > That's why we are here... thanks again for participating. >>>>> > >>>>> > >>>>> > > >>>>> > > It causes me some noise that the FSIP is focused on the API Rest >>>>> layer and >>>>> > > the PR is introducing a new header that could also be used for >>>>> > > Idempotency... We can understand the reasons for having it in the >>>>> Rest >>>>> > > layer. But what about the idempotency that is being used in the >>>>> batch/event >>>>> > > processing? >>>>> > > >>>>> > >>>>> > Sorry for the noise... >>>>> > I'm not really seeing the "focus" on the REST API layer... the >>>>> proposal is >>>>> > kept explicitly free of any assumption which web stack you are >>>>> using... >>>>> > this stuff could even work with Webflux. >>>>> > Introducing a new header? There is not even an implementation for >>>>> > idempotency in this proposal... maybe you are referring to the >>>>> example in >>>>> > the unit test? That stuff (everything in the "sample" package) is >>>>> purely >>>>> > for demonstration. >>>>> > Could you maybe elaborate what problem you see concerning the batch >>>>> > processing? >>>>> > >>>>> > >>>>> > > >>>>> > > I have found the implementation for sync and async commands... >>>>> What about >>>>> > > streaming? Also in the same package there are pipeline, executor, >>>>> router .. >>>>> > > What about notifications? My questions are for getting feedback >>>>> from you if >>>>> > > they are expected/discarded on this new proposed infrastructure. >>>>> > > >>>>> > >>>>> > ... I would add to your list "what about non-blocking"? Well, the >>>>> reason >>>>> > why it's not in my proposal is, because right now we have a classic >>>>> > synchronous blocking execution path in place upstream... and the >>>>> first >>>>> > step would be to get the whole thing gradually to a more performant >>>>> > solution with clearly structured internal API. Another important >>>>> thought >>>>> > here is to avoid/reduce any additional learning curve; the community >>>>> is >>>>> > used to the classic "I call a function and get a result" paradigm of >>>>> > programming; introducing more complex paradigms would need more time >>>>> for >>>>> > adoption. That being said: if the community decided tomorrow to go >>>>> all in >>>>> > for non-blocking, reactive, GRPC, [place your preference here] and >>>>> > bulldozer things then I would happily participate... but my feeling >>>>> is that >>>>> > this is very unlikely. And before anything like this happens we >>>>> would need >>>>> > all these hard-wired references to JSON data structures from the >>>>> business >>>>> > logic services anyway. >>>>> > >>>>> > As for notifications: noted, good point. >>>>> > And concerning feedback: I have no claim here that what I wrote in >>>>> the PR >>>>> > is set in stone... that's why we discuss it here (I hope more people >>>>> join). >>>>> > Things that I didn't have on the radar and that you and others find >>>>> > indispensable can be of course added. >>>>> > >>>>> > >>>>> > > >>>>> > > I think that Apache Camel is a good tool... I don't see it as a >>>>> vendor >>>>> > > lock-in... well If do that (to see it as a vendor lock-in)... then >>>>> what >>>>> > > about spring boot itself? i.e. nowadays we have Quarkus and >>>>> Micronaut... >>>>> > > and I think it is more complex to move some Apache Fineract code >>>>> to these >>>>> > > frameworks or add plugins developed in these frameworks to Apache >>>>> Fineract >>>>> > > runtime. >>>>> > > >>>>> > >>>>> > Again, the idea here is to keep this proposal as self contained as >>>>> possible >>>>> > to avoid having to decide on too many fundamental changes... hence, >>>>> you see >>>>> > no Apache Camel. That being said: I can write you an adapter in >>>>> 10min that >>>>> > runs the whole thing over Camel... but doesn't leak any details >>>>> about Camel >>>>> > being used to the rest of the code base. Don't see what the flaw >>>>> would be >>>>> > here to properly abstract implementation details? >>>>> > >>>>> > Is there a ready to use solution in Spring Boot for command >>>>> processing >>>>> > and/or CQRS? I think not... but let me know if I am missing >>>>> something here. >>>>> > What is out there is Axon which is a complete framework implemented >>>>> ON TOP >>>>> > OF Spring/Boot... but that thing would require us to jump on their >>>>> internal >>>>> > APIs and how they think things should be processed... personally I >>>>> think >>>>> > these guys thought about this subject a bit longer and have a way >>>>> more >>>>> > complete solution... but again, given the need for backwards >>>>> compatibility >>>>> > and the requirement to be not (too) disruptive to any other upstream >>>>> > development: very unlikely that such a fundamental change would >>>>> happen and >>>>> > cause a major refactoring fest. >>>>> > >>>>> > I'm not so sure why Quarkus and Micronaut are mentioned here... yes, >>>>> very >>>>> > nice and capable modern frameworks... but they also have no specific >>>>> notion >>>>> > of command processing, but are generic frameworks like >>>>> Spring/Boot... my >>>>> > intention is not to change any fundamental underlying frameworks >>>>> (give and >>>>> > take they have similar features)... again, I'd like to keep this as >>>>> small >>>>> > and self contained as possible. >>>>> > >>>>> > >>>>> > > >>>>> > > Happy to read your feedback. >>>>> > > >>>>> > > Regards >>>>> > > >>>>> > > Victor >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > El jue, 23 ene 2025 a las 23:40, James Dailey (<jdai...@apache.org >>>>> >) >>>>> > > escribió: >>>>> > > >>>>> > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic >>>>> > >> <chee...@monkeysintown.com> wrote: >>>>> > >> > >>>>> > >> > ... thanks James for the input... I'll try to answer your last >>>>> couple >>>>> > >> of questions from my perspective (read: opinionated... take with >>>>> a pinch of >>>>> > >> salt): >>>>> > >> >>>>> > >> JD: Aleks - thank you. I always learn something from this back >>>>> and >>>>> > >> forth with you. >>>>> > >> > >>>>> > >> > too clever: the current implementation I suggest that anyone >>>>> tries to >>>>> > >> draw a sequence diagram that explains the flow of execution and >>>>> make it fit >>>>> > >> on one page vs the new proposal will most likely contain less >>>>> than a >>>>> > >> handful of lines. You can apply the same if you take lines of >>>>> code as a >>>>> > >> metric... overall the new proposal has less than 50 lines of code >>>>> that are >>>>> > >> relevant (I don't know the number for upstream, but I think it's >>>>> safe to >>>>> > >> say it's more). If we assume that we can achieve the same results >>>>> with less >>>>> > >> code then I think the answer is easy here >>>>> > >> > maintainability: well, see above... the current solution is not >>>>> > >> documented at all and I am pretty sure I am not alone when I saw >>>>> "I really >>>>> > >> can't explain all the steps" (doesn't mean they are not >>>>> necessary); what I >>>>> > >> want to say is that the existing solution would really need a lot >>>>> more >>>>> > >> explanation than just "CQRS", I think that would be a fair >>>>> requirement. >>>>> > >> Admittedly, the new proposal also has no documentation (other >>>>> than the wiki >>>>> > >> page and what I wrote in this message). But: I think if I did >>>>> write it it >>>>> > >> can fit on one page (with diagrams), this module (it's a real >>>>> one) has >>>>> > >> (almost) no external dependencies (other than the frameworks that >>>>> we use >>>>> > >> anyway), it makes no assumption about any of the business logic >>>>> that might >>>>> > >> or might be passing through (existing implementation fails >>>>> already there... >>>>> > >> see CommandWrapper and the various entity IDs that are buried >>>>> there... this >>>>> > >> wrapper class should not be aware of anything it transports)... >>>>> which >>>>> > >> brings me back to the point of less code which is I think from a >>>>> > >> maintenance point of view preferrable >>>>> > >> >>>>> > >> JD: When I say "overly clever" that is in contrast to simplicity >>>>> > >> through elegant design. A favorite quote "There are two ways of >>>>> > >> constructing a software design: One way is to make it so simple >>>>> that >>>>> > >> there are obviously no deficiencies, and the other way is to make >>>>> it >>>>> > >> so complicated that there are no obvious deficiencies. The first >>>>> > >> method is far more difficult." I think if you are aiming for >>>>> > >> something simple enough to have obviously no (or much fewer) >>>>> > >> deficiencies, that is, an improvement. But, could you write some >>>>> > >> documentation about the concept? It should be simple to describe >>>>> "on >>>>> > >> paper", yes? >>>>> > >> >>>>> > >> > >>>>> > >> > Apache Camel: ... disclaimer, I really like that framework and >>>>> used it >>>>> > >> on a ton of occasions. That being said: choosing a framework is a >>>>> > >> commitment pretty much like a vendor lock-in. Depending on how you >>>>> > >> integrate a framework like Camel (this will be more than a JAR >>>>> file and you >>>>> > >> can either hide the fact you use Camel from the rest of your app >>>>> or you >>>>> > >> fully expose it...) upstream means if for some reason it turns >>>>> out that >>>>> > >> Camel is not a good choice or the community doesn't want yet >>>>> another >>>>> > >> dependency then we might find ourselves in a refactoring fest to >>>>> revert >>>>> > >> things. If you look closely in the proposed sources you will see >>>>> that first >>>>> > >> of all there are Java interfaces that propose a contract on how >>>>> to wire >>>>> > >> things together... and there not many... which leaves a lot of >>>>> room for >>>>> > >> actual implementations (Camel or something else). In fact, 3 or 4 >>>>> years ago >>>>> > >> I actually created a drop-in replacement for the upstream >>>>> > >> SynchronousCommandProcessing service and ran Camel behind the >>>>> scenes and >>>>> > >> was actually very happey with the outcome. When I did this there >>>>> were >>>>> > >> basically 2 relevant functions that needed to be taken care of. >>>>> Today there >>>>> > >> is a lot more going on there and I am not so sure if you could >>>>> just drop-in >>>>> > >> Camel effortlessly with the current incarnation of the command >>>>> processing >>>>> > >> service. >>>>> > >> >>>>> > >> JD: Ok. I can buy not wanting another dependency, but only if our >>>>> > >> level of effort is relatively small ongoing. Otherwise we are >>>>> taking >>>>> > >> on code maintenance for our "own thing" when a perfectly suited >>>>> > >> solution is in the same software foundation. >>>>> > >> >>>>> > >> > Asking Apache Camel's community for opinion: well, can't >>>>> hurt... they >>>>> > >> do stuff like this literally every day, so I am pretty sure >>>>> whatever we'll >>>>> > >> exchange with them will be very informative. But that doesn't >>>>> relieve us >>>>> > >> from deciding if you want to go all in on Apache Camel it would >>>>> be anyway a >>>>> > >> good practice to abstract these implementation details away (aka >>>>> hide to >>>>> > >> the rest of Fineract that you are using Camel). If that is the >>>>> case then we >>>>> > >> need a contract (aka Java interface). The one that is there won't >>>>> do it >>>>> > >> anymore... without major rework... and that is the point. The >>>>> proposal >>>>> > >> intends to ensure a gradual non disruptive migration (not open >>>>> heart >>>>> > >> surgery) >>>>> > >> >>>>> > >> JD: Sure, that makes sense, you need to new Java interface... but >>>>> > >> wouldn't it be better to spend a bit of time in design and >>>>> validation >>>>> > >> at this early stage. I think we're talking about a pretty >>>>> significant >>>>> > >> optimization from its location in the stack. Who should reach >>>>> out? >>>>> > >> >>>>> > >> > whitepapers, alternatives: I think the first thing that Google >>>>> or >>>>> > >> ChatGPT searches will tell you is "use an existing CQRS >>>>> framework"... and >>>>> > >> this will most likely show you AxonIQ (a CQRS framework >>>>> implemented with >>>>> > >> Spring/Boot)... but that is then even more of a vendor lock in >>>>> than using a >>>>> > >> more generic solution like Camel... Axon will force us to use >>>>> their >>>>> > >> contracts (internal APIs, Java interfaces etc.), in short: >>>>> refactor fest, >>>>> > >> disruptive. There are other low level "solutions" (like LMAX >>>>> Disruptor) >>>>> > >> that are somewhat in the vicinity of this type of application, >>>>> but require >>>>> > >> work, to my knowledge there is nothing out there we could just >>>>> magically >>>>> > >> drop and use without any refactoring. Disclaimer: in one of the 3 >>>>> drop-in >>>>> > >> implementations of the proposed command processing I am actually >>>>> using LMAX >>>>> > >> Disruptor... its implementation details just don't leak into the >>>>> rest of >>>>> > >> the system >>>>> > >> > Spring Boot 3 compliance: yes (buzzword drop: >>>>> "auto-configuration") >>>>> > >> >>>>> > >> JD: Excellent >>>>> > >> >>>>> > >> > cutting edge: not sure how to read this here... is this meant >>>>> as a >>>>> > >> requirement or as an argument against the adoption of the >>>>> proposal as in >>>>> > >> "too experimental"... as I've written the code I am obviously >>>>> biased so I >>>>> > >> leave that to the community to decide and come up with >>>>> improvements and/or >>>>> > >> alternatives/arguments if someone doesn't agree >>>>> > >> >>>>> > >> JD: Yep. The ambiguity is on purpose - cutting edge can be great >>>>> in >>>>> > >> getting results, or it can make you bleed. >>>>> > >> >>>>> > >> > >>>>> > >> > Let me know if I skipped something, made an error or was not >>>>> clear >>>>> > >> enough. >>>>> > >> >>>>> > >> JD: Very clear. Now, before this code is committed, I would also >>>>> > >> like to be sure we have a sensible way of documenting the >>>>> progress so >>>>> > >> that if we are doing a release, we make note of how much of the >>>>> code >>>>> > >> base is using the new methods. I also think we should discuss >>>>> this in >>>>> > >> context of the next release. (coming up soon). >>>>> > >> >>>>> > >> > >>>>> > >> > On Thu, Jan 23, 2025 at 5:00 PM <jdai...@apache.org> wrote: >>>>> > >> >> >>>>> > >> >> Thanks for bringing this to the list. It looks to be a very >>>>> low level >>>>> > >> (in the stack) and therefore, highly impactful. I was there when >>>>> the >>>>> > >> decision was made to adopt this pattern and >>>>> > >> SynchronousCommandProcessingService as a flexible improvement to >>>>> the >>>>> > >> existing CQRS. I remember asking some questions, but this was and >>>>> is, >>>>> > >> beyond my direct experience. >>>>> > >> >> >>>>> > >> >> >>>>> > >> >>>>> https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure >>>>> > >> >> >>>>> > >> >> What I do know is that we should be deliberate with this >>>>> process, and >>>>> > >> I appreciate your write up on wiki. Definitely other architects >>>>> here >>>>> > >> should take a look. >>>>> > >> >> >>>>> > >> >> At times over the past decisions - it feels to me that we try >>>>> to be >>>>> > >> "too clever", and this creates a problem with maintainability. >>>>> I'd like to >>>>> > >> make sure we understand the alternatives as we dig into this. You >>>>> raised >>>>> > >> Apache Camel as an option - would it be worth it to ask someone >>>>> over in >>>>> > >> that project to comment on this? Is there some whitepaper or >>>>> comparison >>>>> > >> out there between the alternatives available? Is this consistent >>>>> with >>>>> > >> Spring Boot 3 ? Is this on the cutting edge? >>>>> > >> >>>>> > > >>>>> > >>>>> >>>>
