Is it possible that your code is creating EntityListIterator objects and not properly closing them? This may cause connections to stay open until the SqlProcessor objects are finalized. My guess is that the connection leaks are happening before the finalization, and not as a consequence of it, because EntityListIterator are not properly closed, and a db transaction is active when the finalize method attempts to close the connection. Based on the above assumptions, that could be wrong, I don't think that there is an interference of transactions for different tenants, but rather data that you would expect to see is rolled back when the connection is closed before a commit. Please try to investigate in this direction and let me know if you do any progress.
Good luck! Jacopo On Wed, Jul 20, 2016 at 1:51 PM, Jacopo Cappellato < jacopo.cappell...@hotwaxsystems.com> wrote: > Hi Justin, > > On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < > ofbiz-10.04-downst...@fluidnotions.com> wrote: > >> I am really just interested in the architecture, it's no so much an issue >> with any version of ofbiz. (In fact I am working on a system based on >> ofbiz, it would be better to replace it with pure ofbiz, but that is not >> my >> decision to make). I hope you will still be willing to discuss your >> implementation regardless of that fact. >> > > Definitely! > > >> >> Yes org.ofbiz.geronimo.GeronimoTransactionFactory >> calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, >> datasourceInfo.getInlineJdbc()); >> which is implemented >> through >> org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, >> Element) which uses a >> ConcurrentHashMap to store a connection pool per tenant datasource and it >> seems to work very well, the correct pool is provided to the appropriate >> tenant threads. >> >> This is an intermittent issue when testing with 50-100 tenants, >> transactions across tenants seem to collide occasionally. At least that >> seems to be the case, I have stepped through the code in a debugger. >> >> It seem to happen in SqlProcessor finalize method, the gc calls finalize >> and when the connection is closed (returned to the pool) it calls commit >> first and then rolls back because a transaction is in progress as a result >> the connection is leaked and doesn't return to the pool. This is clearly >> seen on a massive scale in jprofiler on the jdbc probe page. >> > > Hmmm... this is an interesting detail, there could be an issue in the > SqlProcessor class or maybe in the way that is used in your framework. > > I will dig into this more and let you know if I find something interesting. > > Jacopo > > >> >> Is it a limitation of the architecture? Is it possible to provide a >> TransactionManger per tenant in ThreadLocal or would that not work? >> >> Or am I missing something here? I realise it's a complex issue, but I am >> trying to stay as close to ofbiz as I can I hope one day we can replace >> the >> framework with a pure ofbiz implementation. I'd rather not make a change >> like this which diverges from ofbiz architecture. How would you solve this >> issue? >> >> >> On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < >> jacopo.cappell...@hotwaxsystems.com> wrote: >> >> > Hi Justin, >> > >> > your understanding of the class implementation is correct and there may >> be >> > a chance that some code may need to be improved, but I am not sure as I >> > could only give a cursory look at the code. >> > By the way, I see that the code in GeronimoTransactionManager calls a >> > "tenant-aware" method in DBCPConnectionFactory and this should provide >> the >> > correct connection to every thread. >> > >> > Could you please elaborate more on your concern? If you could share a >> > usage/failure scenario to recreate the issue it would be great too. >> > >> > Thanks, >> > >> > Jacopo >> > >> > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < >> > ofbiz-10.04-downst...@fluidnotions.com> wrote: >> > >> > > I am trying to understand the database connection pooling in ofbiz 14, >> > > which uses dbcp2. >> > > >> > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there >> is a >> > > static ConcurrentHashMap which provides a ManagedDataSource for each >> > > tenant. >> > > >> > > But when it comes to obtaining a transaction manager reference, it >> looks >> > > like there is one global instance for all tenants. >> > > >> > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory >> > > returns the static reference to a single GeronimoTransactionManager. >> > > >> > > In multi tenant testing overlapping from different tenants seem like >> they >> > > interact with eachother. >> > > >> > > Can you explain how the implementation is supposed to work in with >> multi >> > > tenancy ? >> > > >> > >> > >
