Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Hi Andrea, Nice approach to add logging to Context.java to detect any garbage collected connections. Did that turn up any unclosed connections created by non-customized DSpace code? This made me wonder if there are any cases where db connections are created outside of a Context and if there would be a chance those aren't closed properly. Both shibboleth authentication and the checksum checker seem to be calling the Databasemanager directly without going through Context. Shibboleth Authentication https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/authenticate/ShibAuthentication.java#L961 Checksum Checker https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/checker/BitstreamInfoDAO.java https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/checker/ChecksumHistoryDAO.java https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/checker/ChecksumResultDAO.java https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/checker/ReporterDAO.java best regards Bram -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 315, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.comhttp://atmire.com/website/?q=servicesutm_source=emailfooterutm_medium=emailutm_campaign=braml On Sun, Jan 20, 2013 at 9:56 PM, Andrea Schweer schw...@waikato.ac.nzwrote: Hi all, I'm a bit late to the party^Wthread, but just wanted to comment that my four repositories are all happy with a connection pool size of 50. When I ran into pool exhausted errors, I triple-checked all usage of the Context class in my custom code and found that there were a couple of cases where I created a new Context instance but never committed or aborted it. Such a Context object will hold on to its DB connection until the object is garbage collected and the connection is released in the finalize method. I also added logging output to the finalize method of the Context class so that I get alerted when a Context object with an open DB connection is garbage collected. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnmore_123012 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnnow-d2d___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Hi all, I'm a bit late to the party^Wthread, but just wanted to comment that my four repositories are all happy with a connection pool size of 50. When I ran into pool exhausted errors, I triple-checked all usage of the Context class in my custom code and found that there were a couple of cases where I created a new Context instance but never committed or aborted it. Such a Context object will hold on to its DB connection until the object is garbage collected and the connection is released in the finalize method. I also added logging output to the finalize method of the Context class so that I get alerted when a Context object with an open DB connection is garbage collected. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnmore_123012 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Awesome discussion on a very relevant problem. 2 observations: - Peter's original code worked and was tested - People have much higher values in their connection pools compared to the standard build properties. Combined these two in a jira ticket pull request: https://jira.duraspace.org/browse/DS-1458 But really all kudos go to Peter for the code, André for bringing it back up again, Hilton for the great screenshot on connection usage. -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 315, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.comhttp://atmire.com/website/?q=servicesutm_source=emailfooterutm_medium=emailutm_campaign=braml On Mon, Jan 14, 2013 at 6:39 PM, Peter Dietz pdiet...@gmail.com wrote: Hi André, We've kept the WHEN_EXHAUSTED_GROW code during 1.7, and have just upgraded with it in 1.8, and we are not plagued by the errors and downtime. Since we do allow for an unlimited number of connections, there are times when the site can get slow (heavy load), but it will eventually respond and recover. We don't use Discovery, so no I have no input with regards to that. I don't know if its relevant, but we've also moved PostgreSQL off to its own dedicated server, so atleast the system isn't processing both DSpace/tomcat requests, and DB queries on the same machine. Peter Dietz On Mon, Jan 14, 2013 at 10:38 AM, Hilton Gibson hilton.gib...@gmail.comwrote: You can with: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Disaster_Recovery/System_Monitor On 14 January 2013 15:07, helix84 heli...@centrum.sk wrote: Hilton, that's a wonderful graph exactly illustrating the problem (the spike in idle in transaction connections). I wish I had such monitoring set up :) -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow - 350+ hours of step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122812___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Dear all, Have Peter's code solved the problem, could anyone using DSpace XMLUI 1.7.1 + Discovery give an update? Also, I noticed that after this Cannot get a connection, java starts to consume 100% of processor and the user interface stops responding, has anyone faced the same problem? Regards André N. Assada 2012/7/19 helix84 heli...@centrum.sk On Thu, Jul 19, 2012 at 6:08 AM, Benjamin Ryan benjamin.r...@manchester.ac.uk wrote: Helix, Do you have any details of XOAI please? Sure, it's a new implementation of the OAI webapp from scratch by Lyncode, also known as OAI 2.0. In fact xoai-common is the underlying generic OAI library (like OAICat was for the old webapp) and OAI 2.0 is the data provider (glue) for DSpace. There's code for 3.0 that replaces the old OAI module and an addon for 1.8 which you can install alongside as a separate webapp. wiki docs: https://wiki.duraspace.org/display/DSPACE/OAI+2.0 Jira issue: https://jira.duraspace.org/browse/DS-1202 Pull Request https://github.com/DSpace/DSpace/pull/37 Source code on GitHub (based on git master, i.e. 3.0) in branch dspace-with-xoai: https://github.com/lyncode/DSpace Source code packaged as addon for 1.8 + installation docs: http://www.lyncode.com/dspace/addons/xoai/XOAI.zip Any testing, comments and suggestions are welcome, you can use the Jira comments or mailing list. Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
On Mon, Jan 14, 2013 at 12:26 PM, André andre.ass...@usp.br wrote: Have Peter's code solved the problem, could anyone using DSpace XMLUI 1.7.1 + Discovery give an update? Also, I noticed that after this Cannot get a connection, java starts to consume 100% of processor and the user interface stops responding, has anyone faced the same problem? Yes, this occasionally happened to me, both on 1.8.x and 3.0. I think it happened to me only once on 3.0, where OAI no longer consumes DB connections, so I recommend upgrading as a workaround. There is no solution yet because we still have little information about the problem - it's hard to pin down the conditions reproducing it reliably. Regards, ~~helix84 -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
BTW there is work going on, tentatively targetted at DSpace 4.0, which changes completely how the DB access layer works, so it's possible this problem will be solved as a side-effect, in about a year. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Try reviewing this step: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Prepare_Ubuntu/S06, it may give you some pointers. Also see: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations On 14 January 2013 13:26, André andre.ass...@usp.br wrote: Dear all, Have Peter's code solved the problem, could anyone using DSpace XMLUI 1.7.1 + Discovery give an update? Also, I noticed that after this Cannot get a connection, java starts to consume 100% of processor and the user interface stops responding, has anyone faced the same problem? Regards André N. Assada 2012/7/19 helix84 heli...@centrum.sk On Thu, Jul 19, 2012 at 6:08 AM, Benjamin Ryan benjamin.r...@manchester.ac.uk wrote: Helix, Do you have any details of XOAI please? Sure, it's a new implementation of the OAI webapp from scratch by Lyncode, also known as OAI 2.0. In fact xoai-common is the underlying generic OAI library (like OAICat was for the old webapp) and OAI 2.0 is the data provider (glue) for DSpace. There's code for 3.0 that replaces the old OAI module and an addon for 1.8 which you can install alongside as a separate webapp. wiki docs: https://wiki.duraspace.org/display/DSPACE/OAI+2.0 Jira issue: https://jira.duraspace.org/browse/DS-1202 Pull Request https://github.com/DSpace/DSpace/pull/37 Source code on GitHub (based on git master, i.e. 3.0) in branch dspace-with-xoai: https://github.com/lyncode/DSpace Source code packaged as addon for 1.8 + installation docs: http://www.lyncode.com/dspace/addons/xoai/XOAI.zip Any testing, comments and suggestions are welcome, you can use the Jira comments or mailing list. Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
On Mon, Jan 14, 2013 at 1:22 PM, Hilton Gibson hilton.gib...@gmail.com wrote: Try reviewing this step: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Prepare_Ubuntu/S06, it may give you some pointers. Also see: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations Hi Hilton, unfortunately, I don't see anything there that would prevent the problem. My configuration is very similar: max_connections = 100 kernel.shmmax = 536870912 kernel.shmall = 536870912 The immediate problem manifests by most DB connections being in IDLE in transaction state when you view postgres process list. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
I think I have setup max connections to 300 and unlimited idle connections for both the PostgreSQL DB and in the DSpace config. So far never had a DB connection problem. I suppose it depends on the visitor density ie: visits per minute. On 14 January 2013 14:34, helix84 heli...@centrum.sk wrote: max_connections = 100 kernel.shmmax = 536870912 kernel.shmall = 536870912 -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Here are my db options from dspace.cfg of that instance (default except raised db.maxconnections): # Maximum number of DB connections in pool db.maxconnections = 90 # Maximum time to wait before giving up if all connections in pool are busy (milliseconds) db.maxwait = 5000 # Maximum number of idle connections in pool (-1 = unlimited) db.maxidle = -1 # Determine if prepared statement should be cached. (default is true) db.statementpool = true Hilton, what do you use for db.maxidle? I'm wondering if that could be what makes a difference. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Also here is a graph. On 14 January 2013 14:48, helix84 heli...@centrum.sk wrote: Here are my db options from dspace.cfg of that instance (default except raised db.maxconnections): # Maximum number of DB connections in pool db.maxconnections = 90 # Maximum time to wait before giving up if all connections in pool are busy (milliseconds) db.maxwait = 5000 # Maximum number of idle connections in pool (-1 = unlimited) db.maxidle = -1 # Determine if prepared statement should be cached. (default is true) db.statementpool = true Hilton, what do you use for db.maxidle? I'm wondering if that could be what makes a difference. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za attachment: postgres_connections_ALL-year.png-- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
On Mon, Jan 14, 2013 at 12:26 PM, André andre.ass...@usp.br wrote: Have Peter's code solved the problem, could anyone using DSpace XMLUI 1.7.1 + Discovery give an update? BTW André, sorry, I didn't really answer your question because I didn't notice it pertained to the previous conversation. So no, I haven't verified whether Peter's code solved the problem, I'm using stock code in this respect. On Mon, Jan 14, 2013 at 2:01 PM, Hilton Gibson hilton.gib...@gmail.com wrote: Also here is a graph. Hilton, that's a wonderful graph exactly illustrating the problem (the spike in idle in transaction connections). I wish I had such monitoring set up :) But I wonder what the other spike is (idle transactions). Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
No problem, Helix, and thank you all a lot for all the info, it gave a good insight of the problem anyways. Regards, André N. Assada 2013/1/14 helix84 heli...@centrum.sk On Mon, Jan 14, 2013 at 12:26 PM, André andre.ass...@usp.br wrote: Have Peter's code solved the problem, could anyone using DSpace XMLUI 1.7.1 + Discovery give an update? BTW André, sorry, I didn't really answer your question because I didn't notice it pertained to the previous conversation. So no, I haven't verified whether Peter's code solved the problem, I'm using stock code in this respect. On Mon, Jan 14, 2013 at 2:01 PM, Hilton Gibson hilton.gib...@gmail.com wrote: Also here is a graph. Hilton, that's a wonderful graph exactly illustrating the problem (the spike in idle in transaction connections). I wish I had such monitoring set up :) But I wonder what the other spike is (idle transactions). Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
You can with: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Disaster_Recovery/System_Monitor On 14 January 2013 15:07, helix84 heli...@centrum.sk wrote: Hilton, that's a wonderful graph exactly illustrating the problem (the spike in idle in transaction connections). I wish I had such monitoring set up :) -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Hi André, We've kept the WHEN_EXHAUSTED_GROW code during 1.7, and have just upgraded with it in 1.8, and we are not plagued by the errors and downtime. Since we do allow for an unlimited number of connections, there are times when the site can get slow (heavy load), but it will eventually respond and recover. We don't use Discovery, so no I have no input with regards to that. I don't know if its relevant, but we've also moved PostgreSQL off to its own dedicated server, so atleast the system isn't processing both DSpace/tomcat requests, and DB queries on the same machine. Peter Dietz On Mon, Jan 14, 2013 at 10:38 AM, Hilton Gibson hilton.gib...@gmail.comwrote: You can with: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Disaster_Recovery/System_Monitor On 14 January 2013 15:07, helix84 heli...@centrum.sk wrote: Hilton, that's a wonderful graph exactly illustrating the problem (the spike in idle in transaction connections). I wish I had such monitoring set up :) -- *Hilton Gibson* Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://scholar.sun.ac.za http://ar1.sun.ac.za http://aj1.sun.ac.za -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
On Thu, Jul 19, 2012 at 6:08 AM, Benjamin Ryan benjamin.r...@manchester.ac.uk wrote: Helix, Do you have any details of XOAI please? Sure, it's a new implementation of the OAI webapp from scratch by Lyncode, also known as OAI 2.0. In fact xoai-common is the underlying generic OAI library (like OAICat was for the old webapp) and OAI 2.0 is the data provider (glue) for DSpace. There's code for 3.0 that replaces the old OAI module and an addon for 1.8 which you can install alongside as a separate webapp. wiki docs: https://wiki.duraspace.org/display/DSPACE/OAI+2.0 Jira issue: https://jira.duraspace.org/browse/DS-1202 Pull Request https://github.com/DSpace/DSpace/pull/37 Source code on GitHub (based on git master, i.e. 3.0) in branch dspace-with-xoai: https://github.com/lyncode/DSpace Source code packaged as addon for 1.8 + installation docs: http://www.lyncode.com/dspace/addons/xoai/XOAI.zip Any testing, comments and suggestions are welcome, you can use the Jira comments or mailing list. Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Hi All, I'm curious if anyone has had the unfortunate case of being plagued by database pool errors? Our environment is Tomcat (5.5) + Postgres (8.4) + XMLUI (1.7.1) An example of the error we had been getting is: /dspace/log/dspace.log.2012-07-01:2012-07-01 12:25:59,627 ERROR org.dspace.app.xmlui.aspect.general.AuthenticatedSelector @ Error selecting based on authentication status: Cannot get a connection, pool error Timeout waiting for idle object From my understanding is that DSpace will use a database connection pool, which is configurable to have some 30 open connections between the JVM, and postgres. When a new request comes in, that requires interaction with the database, it will re-use an existing already-open database connection, taking it from the pool, make the query, and then return the connection back to the pool. The problem must lie in that somehow more requests come in at a certain moment, which is more than the number of available connections in the pool, and the default behavior is to wait/block until a connection becomes available. There is a timeout for how long it will wait for a connection to become available, after which time, it errors out with the above error. Reasons why all the connections in the pool are in use could be that the system is very busy, and its just using all the connections, or as I suspect, there is a bug that is prohibiting some connections from being returned back to the pool. I can't quite put my finger on precisely what could be wrong. Anyways, I have made a change to my database pool settings in DSpace, and I've noticed that since we've made this change, we haven't seem the error occur again. dspace-api/src/main/java/org/dspace/storage/rdbms/DataSourceInit.java https://github.com/osulibraries/DSpaceOSUKB/commit/c627ebdb738efbf46ba7a338180099fa38b96a66.diff // Create object pool ObjectPool connectionPool = new GenericObjectPool(null, // PoolableObjectFactory -// - set below -maxConnections, // max connections -GenericObjectPool.*WHEN_EXHAUSTED_BLOCK*, maxWait, // don't - // block -// more than 5 -// seconds -maxIdle, // max idle connections (unlimited) -true, // validate when we borrow connections from pool -false // don't bother validation returned connections +maxConnections, // max connections +GenericObjectPool.*WHEN_EXHAUSTED_GROW*, // What to do when pool is exhausted +maxWait,// don't block more than 5 seconds +maxIdle,// max idle connections (unlimited) +true, // validate on connection borrow ? +true// validate on connection return ? Basically, the biggest change in there is the WHEN_EXHAUSTED action. The default is to use the strategy of WHEN_EXHAUSTED_BLOCK, and I've changed that locally to WHEN_EXHAUSTED_GROW, which essentially nullifies any max-number-of-connections limit that I've set. But, atleast we don't have the problem where a submitter would randomly become plagued with not being able to use the system. As I've said, since we've made this change, we've been two weeks error free (on this issue). I'm wary to claim victory on one of these sneaky bugs, but I was wondering if anyone else has run into this issue, and perhaps has found some action steps to resolve it. Peter Dietz -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Hi Peter, I was haunted by this same bug almost every time when OAI was under load and in some other rare cases, too. But I gave up on solving it in favor of XOAI which doesn't have this problem because it uses Solr instead of database (it's configurable to use database if you want, but I haven't encountered the problem, but then I didn't stress test it, either). Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object
Helix, Do you have any details of XOAI please? Regards, Ben -- Dr Ben Ryan Jorum Technical Coordinator (Services) 5.12 Roscoe Building The University of Manchester Oxford Road Manchester M13 9PL Tel: 0160 275 6039 E-mail: benjamin.r...@manchester.ac.uk -- -Original Message- From: helix84 [mailto:heli...@centrum.sk] Sent: 18 July 2012 23:01 To: Peter Dietz Cc: Dspace Tech Subject: Re: [Dspace-tech] Database Pool Error: Cannot get a connection, pool error. Timeout waiting for idle object Hi Peter, I was haunted by this same bug almost every time when OAI was under load and in some other rare cases, too. But I gave up on solving it in favor of XOAI which doesn't have this problem because it uses Solr instead of database (it's configurable to use database if you want, but I haven't encountered the problem, but then I didn't stress test it, either). Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
