Re: DIH: Exception with Too many connections
looks like you are not being able to connect to database , pls see if you get similar exception when you try to connect from other clients On Tue, May 31, 2011 at 3:01 PM, tiffany tiffany.c...@future.co.jp wrote: Hi all, I'm using DIH and getting the following error. My Solr version is Solr3.1. = ... Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up. at sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) at com.mysql.jdbc.Util.getInstance(Util.java:381) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2364) at com.mysql.jdbc.ConnectionImpl.init(ConnectionImpl.java:781) at com.mysql.jdbc.JDBC4Connection.init(JDBC4Connection.java:46) at sun.reflect.GeneratedConstructorAccessor94.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:352) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:284) at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:161) at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:128) at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:363) at org.apache.solr.handler.dataimport.JdbcDataSource.access$200(JdbcDataSource.java:39) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.init(JdbcDataSource.java:240) ... 11 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: Too many connections at sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) at com.mysql.jdbc.Util.getInstance(Util.java:381) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1104) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2292) ... 24 more = My dataSource setting is something like this: dataSource type=JdbcDataSource driver=com.mysql.jdbc.Driver url=jdbc:mysql://database01/test?autoReconnect=true user=xxx password=xxx batchSize=-1 / Any idea to solve this problem? Thank you! -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-Exception-with-Too-many-connections-tp3005213p3005213.html Sent from the Solr - User mailing list archive at Nabble.com. -- Chandan Tamrakar * *
Re: DIH: Exception with Too many connections
Tiffany, On Tue, May 31, 2011 at 11:16 AM, tiffany tiffany.c...@future.co.jp wrote: Any idea to solve this problem? in Addition to Chandan: Check your mysql process list and have a look what is displayed there Regards Stefan
Re: DIH: Exception with Too many connections
Thanks for your reply, Chandan. Here is the additional information. I'm also using the multi-core function, and I run the delta-import command in parallel due to saving the running time. If I don't run in parallel, it works fine. Each core accesses to the same database server but different schema. So, I don't know if I should change something in my database server side or I can adjust something at the Solr side by adding some kind of property. Tiffany -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-Exception-with-Too-many-connections-tp3005213p3005313.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH: Exception with Too many connections
Thanks Stefan! I executed the SHOW PROCESSLIST; command. (Is it what you mean? I've never tried it before...) It seems that when I executed one delta-import command, several threads were inserted into the table and removed after commit. Also it looks like the number of threads are pretty much equal to the number of entity in my db-data-config.xml. So, if the number of threads in the process list is larger than max_connections, I would get the too many connections error. Am I thinking the right way? If it is right, maybe I should think of the commit timing, changing the number of max_connections, and/or some other ways... If there are any other idea, please let me know =) Thanks a lot! -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-Exception-with-Too-many-connections-tp3005213p3005401.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH: Exception with Too many connections
Tiffany, On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: I executed the SHOW PROCESSLIST; command. (Is it what you mean? I've never tried it before...) Exactly this, yes :) On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: So, if the number of threads in the process list is larger than max_connections, I would get the too many connections error. Am I thinking the right way? Yepp, right On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: If it is right, maybe I should think of the commit timing, changing the number of max_connections, and/or some other ways... You may lift the allowed Number of Connections for the MySQL-Server? Or, of course - if possible - tweak your SOLR-Settings, correct Regards Stefan
Re: DIH: Exception with Too many connections
Hi You might also check the 'max_user_connections' settings too if you have that set: # Maximum number of connections, and per user max_connections = 2048 max_user_connections = 2048 http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html Cheers François On May 31, 2011, at 7:39 AM, Stefan Matheis wrote: Tiffany, On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: I executed the SHOW PROCESSLIST; command. (Is it what you mean? I've never tried it before...) Exactly this, yes :) On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: So, if the number of threads in the process list is larger than max_connections, I would get the too many connections error. Am I thinking the right way? Yepp, right On Tue, May 31, 2011 at 12:45 PM, tiffany tiffany.c...@future.co.jp wrote: If it is right, maybe I should think of the commit timing, changing the number of max_connections, and/or some other ways... You may lift the allowed Number of Connections for the MySQL-Server? Or, of course - if possible - tweak your SOLR-Settings, correct Regards Stefan
RE: DIH: Exception with Too many connections
Hi, There is existing bug in DataImportHandler described (and patched) at https://issues.apache.org/jira/browse/SOLR-2233 It is not used in a thread safe manner, and it is not appropriately closed reopened (why?); and new connection is opened unpredictably. It may cause Too many connections even for huge SQL-side max_connections. If you are interested, I can continue work on SOLR-2233. CC: dev@lucene (is anyone working on DIH improvements?) Thanks, Fuad Efendi http://www.tokenizer.ca/ -Original Message- From: François Schiettecatte [mailto:fschietteca...@gmail.com] Sent: May-31-11 7:44 AM To: solr-user@lucene.apache.org Subject: Re: DIH: Exception with Too many connections Hi You might also check the 'max_user_connections' settings too if you have that set: # Maximum number of connections, and per user max_connections = 2048 max_user_connections = 2048 http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html Cheers François So, if the number of threads in the process list is larger than max_connections, I would get the too many connections error. Am I thinking the right way?
RE: DIH: Exception with Too many connections
Stephan, Your advice (check the process list) gave me an important clue for my solution. I changed my database connection to the slave instead of master, so that I can use more threads. Thank you very much! *** François, My setting is the default value: max_connections = 151 max_user_connections = 0 I will think of changing the max_connections when I increase the number of cores. Thanks! *** Fuad, So far, I can handle DIH with my setting. Thanks for letting me know! Regards, Tiffany -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-Exception-with-Too-many-connections-tp3005213p3009206.html Sent from the Solr - User mailing list archive at Nabble.com.
