Hi Graham, Thanks you for your comment and question. Trafodion does have a light version that we use for dev. We call it local_hadoop, and it fits in single dev boxes. This will perfectly fit this use cases, Hope this helps, Eric
From: Ebenezer Graham <[email protected]> Sent: Tuesday, June 4, 2019 7:53 AM To: Dev <[email protected]> Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer? External Hi Eric, Thanks for this interesting and insightful proposal. Please expound on how to use Trafodion in component tests and CI/CD. If you noticed, all the services have a component-test package aside from the unit tests. This is used to test the service during development as well as for CI/CD - the data from the component tests is persisted to an embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a similar CQRS pattern as will be used in production. How will this work when using Trafodion? Does Trafodion have a light-weight version for this use case? If not, what is a typical approach for this? I have looked at the specs for a basic Trafodion setup and its quite resource intensive. Looking forward to your response At your service, Ebenezer Graham On Fri, 31 May 2019 at 02:45, Eric Owhadi <[email protected]<mailto:[email protected]>> wrote: Hi James, Thanks for taking the time to answer, Please find bellow inline answers to your questions. What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries? >>> I am not yet familiar with the code, so can’t comment on this. What I can >>> say is that Trafodion uses a standard jdbc driver, and other than dealing >>> with minor SQL dialect differences that may exist (like any popular RDBMS >>> dialect minor differences), the adaptation to Trafodion should be minimal >>> (depending on how the code is structured – understand if the SQL statement >>> are well separated vs have to hunt for them in many places in the code). Having said that, the statement is valid for just replacing RDBMS calls to trafodion. Then replacing Cassandra calls to trafodion should also be easy. However, if we want to replace some of current CQRS commands from Async to Sync, the code change will be more involved, as I assume the whole thing about doing “light check” before performing the async command, and the un-necessary error handling involved at the use case level (what happen if an async command fail, and UI thought it went OK) should be optionally removed. Instead classic RDBMS transaction commit/rollback would be done. Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available? >>> general DB skill. Easier than current solution, as standard SQL is easier >>> to find than standard SQL + Cassandra special. Is the trajectory of this technology vs noSql clear? >>> Yes. It’s purpose is to add back everything that was given up (ACID) from >>> classic RDBMS to achieve horizontal scalability. Now you have a full >>> classic RDBMS, but capable of handling huge volume, with scale out share >>> nothing architecture. You've made a good case, what's the counter? >>> The counter argument would be maturity. MySQL, Postgres or MariaDB, given >>> their wide adoption, benefit from more battle testing than Trafodion. >>> Having said that, and given the recent commitment to core banking, EsgynDB >>> will be battle tested for core Banking, while the other DB are not >>> typically seen in the “classic” Core Banking sector. By the time >>> fineract-cn will be feature complete, I think the maturity argument will >>> have seriously faded out. What about Apache Ignite? >>> Apache Ignite is a different beast. Classified as “in memory DB”. I have >>> no experience with it, but can say that what is needed is a persistence >>> layer, not an in memory db to speed up analytical queries. Therefore I have >>> hard time seeing a fit with Ignite. Is there an independent review of Trafodion? >>> Not that I know of. Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech? >>> I would not use Trafodion in production, given I would never go in >>> production without support contract on the database. Currently only Esgyn >>> Corporation provide a Trafodion based production ready product. If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks? >>> On my free cycles, I will study fineract-cn code and analyze effort / >>> prototype something. Best regards, Eric Owhadi From: James Dailey <[email protected]<mailto:[email protected]>> Sent: Thursday, May 30, 2019 4:26 PM To: Dev <[email protected]<mailto:[email protected]>> Cc: Eric Owhadi <[email protected]<mailto:[email protected]>> Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer? External Eric, First, thank you very much. This is a really interesting concept and proposal for a shift in architecture. I have some questions: What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries? Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available? Is the trajectory of this technology vs noSql clear? What about Apache Ignite? Is there an independent review of Trafodion? You've made a good case, what's the counter? Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech? If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks? Thanks, Jdailey67 On Thu, May 30, 2019, 1:48 AM Eric Owhadi <[email protected]<mailto:[email protected]>> wrote: Hello Fineracteers, I discovered Fineract and specially Fineract-cn last week, and am very impressed by the future potential of the project, both to deliver on its original social mission but also to revolutionize commercial FinTech sector the same way the big data wave caught up when Hadoop was open sourced. The architecture [Cloud Native/micro-services] is at the forefront of what a modern application should look like, and the decision to start from scratch and don’t carry the burden of legacy was a bold but very smart move imho. There is however, one area that might be interesting to challenge: the CQRS and the dual storage engine needed to provide capability for async command execution (and its draw back on error handling from front end) and sync query accessing the traditional DBMS with its scalability shortfall. The current implementation looks optimal when it was picked (I assume when fineract-cn was still mifos io), because there was no open source “NewSQL” solution available. Using classic RDBMS with known scalability limitation, coupled with NoSQL eventual consistency, no ACID, no multicolumn/table transaction, no join or grouping, given up to gain stellar horizontal scalability was a smart move. However, in 2017, Apache released Trafodion as Top Level Project. It could replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is fully ACID, supports multicolumn/multi table transactions, joins, group by, is fully horizontally scalable and support all the bells and whistle you would expect from a classic RDBMS. It can be classified as NewSQL, basically claiming back all the things given up by NoSql in exchange for scalability. In addition, the company backing it up (Esgyn Corp) have recently announced that its commercial product (EsgynDB, derived from Trafodion) is being used for Core Banking in 2 major Banks in China, struggling with the ever growing mobile payment volume that traditional scale up databases cannot keep up with. In addition Esgyn provides a DB as a service offering, cloud hosted fully managed (Esgyn Strato), giving the option to potential fineract-cn hosting company to “outsource” the persistence layer management and focus on the rest. Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can think of the following advantages: * Simplify persistence layer ecosystem from 2 DB to 1 -> back to classic RDBMS but with full scale out capability MPP style. * Still Apache license compatible * Persistence layer ready to support challenging analytical workload (reporting use cases? - thanks for reclaiming Join/group-by at scale). * Remove the reticence of decision maker to use NoSQL in Core Banking * Performance over Cassandra [on paper, even if Trafodion was used only as a drop in replacement to Cassandra for async commands, the use of Trafodion “Align Format” feature should demonstrate higher performance for journaling, as Trafodion would write 1 BigTable cell per row, while Cassandra or Native Hbase would write 1 X nb Column cells per row]. * With a solid company already showing its commitment to core banking backing the Apache project Hope this makes sense? Eric Owhadi, https://www.linkedin.com/in/eric-owhadi-3173058/ Principal Engineer at Esgyn Corporation. message resent from personal email, the first tentative did not make it...
