Hi All,
Calling all JDBC pooling experts. So, during the stress-testing of the
REST-API many months ago, I've had jmeter hit REST full throttle for hours,
fixed bugs in REST to support reliability and performance, and now it
supports about a million requests per hour, depending on endpoint.
However, I realized that I was doing something clever, or not-clever
depending on perspective. REST doesn't participate in the database pool
(getting a context, doing its thing, then returning the context at the end
of the method). Instead I had stored a private static copy of the context,
and never return it. Therefore, each endpoint holds its own connection,
doesn't go through the overhead of getting a new context, and doesn't go
through the overhead of closing the connection. This is quicker, but I've
just discovered that it causes rest to not have real-time data, and it
won't work when we add AuthN to rest.
However, most importantly, I don't think that the DatabaseManager /
DataSource / pool / context-stuff, properly supports this level of grabbing
a new connection, and releasing the connection. My jmeter test hits 7
endpoints, full throttle, and never stops.
The current 4.0 version of REST uses "keep connection". It performs well,
0% error, 50-200 requests/per second, about a million per hour. However, by
holding on to a static database connection, it can get
less-than-real-time-data.
if(context == null) {
context = new org.dspace.core.Context();
}
... do your thing
If I change all the endpoints to use the pool, each request grabbing a new
context, does its things, then closes the context, I get 80-100% error rate.
The code for new context / close context looks like:
context = new org.dspace.core.Context();
... do your thing
context.complete();
Errors are of the flavor:
ERROR org.dspace.rest.CommunitiesResource @ This statement has been closed.
ERROR org.dspace.rest.CommunitiesResource @ Cannot get a connection, pool
error Timeout waiting for idle object
So, to me, it looks like the database connection pool isn't working
correctly, or isn't supposed to be used like this.
So, if anyone has any ideas on what can be done, that would be greatly
appreciated. I'm tempted to dig into the DatabaseManager classes of DSpace,
but that is terrifying. I do suspect that proper use of things like JDBC
pools ought to support heavy usage. We are using JDBC 1.4, which is the
latest for Java6, JDBC 2.0 exists and requires Java7. I'm wondering if
perhaps we have a bug in the database manager code that uses a regular
database Connection, and not the PoolableConnection. These details are
beyond me, but a webapp should be able to have a good
internal-dspace-framework to stand upon. I don't know if this is where ORM
or something like jooq would help.
Thanks
Peter Dietz
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
