Hi Eric, Thanks for this interesting and insightful proposal.
Please expound on how to use Trafodion in component tests and CI/CD. If you noticed, all the services have a component-test package aside from the unit tests. This is used to test the service during development as well as for CI/CD - the data from the component tests is persisted to an embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a similar CQRS pattern as will be used in production. How will this work when using Trafodion? Does Trafodion have a light-weight version for this use case? If not, what is a typical approach for this? I have looked at the specs for a basic Trafodion setup and its quite resource intensive. Looking forward to your response *At your service,* *Ebenezer Graham* On Fri, 31 May 2019 at 02:45, Eric Owhadi <[email protected]> wrote: > Hi James, > > Thanks for taking the time to answer, > > Please find bellow inline answers to your questions. > > > > What would be the impact on current efforts to remove current dependencies > on non Apache license compatible data persistent layers/libraries? > > >>> I am not yet familiar with the code, so can’t comment on this. What I > can say is that Trafodion uses a standard jdbc driver, and other than > dealing with minor SQL dialect differences that may exist (like any popular > RDBMS dialect minor differences), the adaptation to Trafodion should be > minimal (depending on how the code is structured – understand if the SQL > statement are well separated vs have to hunt for them in many places in the > code). > > Having said that, the statement is valid for just replacing RDBMS calls to > trafodion. Then replacing Cassandra calls to trafodion should also be easy. > However, if we want to replace some of current CQRS commands from Async to > Sync, the code change will be more involved, as I assume the whole thing > about doing “light check” before performing the async command, and the > un-necessary error handling involved at the use case level (what happen if > an async command fail, and UI thought it went OK) should be optionally > removed. Instead classic RDBMS transaction commit/rollback would be done. > > > > Once implemented, is this more, less, same to maintain and add > functionality? Would it require a dev to map the data logical level (ORM) > more intensively? Is there a different skill set needed with this than > current concept and are those skills generally available? > > >>> general DB skill. Easier than current solution, as standard SQL is > easier to find than standard SQL + Cassandra special. > > > > Is the trajectory of this technology vs noSql clear? > > >>> Yes. It’s purpose is to add back everything that was given up (ACID) > from classic RDBMS to achieve horizontal scalability. Now you have a full > classic RDBMS, but capable of handling huge volume, with scale out share > nothing architecture. > > > > You've made a good case, what's the counter? > > >>> The counter argument would be maturity. MySQL, Postgres or MariaDB, > given their wide adoption, benefit from more battle testing than Trafodion. > Having said that, and given the recent commitment to core banking, EsgynDB > will be battle tested for core Banking, while the other DB are not > typically seen in the “classic” Core Banking sector. By the time > fineract-cn will be feature complete, I think the maturity argument will > have seriously faded out. > > > > What about Apache Ignite? > > >>> Apache Ignite is a different beast. Classified as “in memory DB”. I > have no experience with it, but can say that what is needed is a > persistence layer, not an in memory db to speed up analytical queries. > Therefore I have hard time seeing a fit with Ignite. > > > > Is there an independent review of Trafodion? > > >>> Not that I know of. > > > > Is the open source version useable or does this require use of the EsgynDB > in production? Are other providers using the tech? > > >>> I would not use Trafodion in production, given I would never go in > production without support contract on the database. Currently only Esgyn > Corporation provide a Trafodion based production ready product. > > > > > > If the project devs can come to some conclusion on this, what's the level > of effort? How should we do think about the size of this effort. & do you > think... is there someone available to work on this project, perhaps from > one of those banks? > > >>> On my free cycles, I will study fineract-cn code and analyze effort / > prototype something. > > > > Best regards, > > Eric Owhadi > > > > *From:* James Dailey <[email protected]> > *Sent:* Thursday, May 30, 2019 4:26 PM > *To:* Dev <[email protected]> > *Cc:* Eric Owhadi <[email protected]> > *Subject:* Re: Would it be interesting to use Apache "Trafodion" as an > option for persistence layer? > > > > External > > Eric, > > > > First, thank you very much. > > > > This is a really interesting concept and proposal for a shift in > architecture. I have some questions: > > > > What would be the impact on current efforts to remove current dependencies > on non Apache license compatible data persistent layers/libraries? > > > > Once implemented, is this more, less, same to maintain and add > functionality? Would it require a dev to map the data logical level (ORM) > more intensively? Is there a different skill set needed with this than > current concept and are those skills generally available? > > > > Is the trajectory of this technology vs noSql clear? What about Apache > Ignite? Is there an independent review of Trafodion? You've made a good > case, what's the counter? > > > > Is the open source version useable or does this require use of the EsgynDB > in production? Are other providers using the tech? > > > > If the project devs can come to some conclusion on this, what's the level > of effort? How should we do think about the size of this effort. & do you > think... is there someone available to work on this project, perhaps from > one of those banks? > > > > Thanks, > > Jdailey67 > > > > > > On Thu, May 30, 2019, 1:48 AM Eric Owhadi <[email protected]> > wrote: > > Hello Fineracteers, > > I discovered Fineract and specially Fineract-cn last week, and am very > impressed by the future potential of the project, both to deliver on its > original social mission but also to revolutionize commercial FinTech sector > the same way the big data wave caught up when Hadoop was open sourced. > > The architecture [Cloud Native/micro-services] is at the forefront of what > a modern application should look like, and the decision to start from > scratch and don’t carry the burden of legacy was a bold but very smart move > imho. > > There is however, one area that might be interesting to challenge: the > CQRS and the dual storage engine needed to provide capability for async > command execution (and its draw back on error handling from front end) and > sync query accessing the traditional DBMS with its scalability shortfall. > > > > The current implementation looks optimal when it was picked (I assume when > fineract-cn was still mifos io), because there was no open source “NewSQL” > solution available. Using classic RDBMS with known scalability limitation, > coupled with NoSQL eventual consistency, no ACID, no multicolumn/table > transaction, no join or grouping, given up to gain stellar horizontal > scalability was a smart move. > > > > However, in 2017, Apache released Trafodion as Top Level Project. It could > replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS > built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is > fully ACID, supports multicolumn/multi table transactions, joins, group by, > is fully horizontally scalable and support all the bells and whistle you > would expect from a classic RDBMS. It can be classified as NewSQL, > basically claiming back all the things given up by NoSql in exchange for > scalability. In addition, the company backing it up (Esgyn Corp) have > recently announced that its commercial product (EsgynDB, derived from > Trafodion) is being used for Core Banking in 2 major Banks in China, > struggling with the ever growing mobile payment volume that traditional > scale up databases cannot keep up with. In addition Esgyn provides a DB as > a service offering, cloud hosted fully managed (Esgyn Strato), giving the > option to potential fineract-cn hosting company to “outsource” the > persistence layer management and focus on the rest. > > > > Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can > think of the following advantages: > > - Simplify persistence layer ecosystem from 2 DB to 1 -> back to > classic RDBMS but with full scale out capability MPP style. > - Still Apache license compatible > - Persistence layer ready to support challenging analytical workload > (reporting use cases? - thanks for reclaiming Join/group-by at scale). > - Remove the reticence of decision maker to use NoSQL in Core Banking > - Performance over Cassandra [on paper, even if Trafodion was used > only as a drop in replacement to Cassandra for async commands, the use of > Trafodion “Align Format” feature should demonstrate higher performance for > journaling, as Trafodion would write 1 BigTable cell per row, while > Cassandra or Native Hbase would write 1 X nb Column cells per row]. > - With a solid company already showing its commitment to core banking > backing the Apache project > > > > Hope this makes sense? > > Eric Owhadi, > > https://www.linkedin.com/in/eric-owhadi-3173058/ > > Principal Engineer at Esgyn Corporation. > > message resent from personal email, the first tentative did not make it... > >
