Hi Emmanuel / Robert On Mon, 19 May 2008 13:02:06 +0200 Emmanuel Cecchet <[EMAIL PROTECTED]> wrote:
> Hi, > > > Recently we have deployed Sequoia into a new data center with the > > following configuration. > > > > 2 controllers, 2 backends per conntroller. Web application access > > Sequoia controller via ASA, allowing port 25322 TCP/UDP. Controllers > > access Postgresql Database via ASA, allowing port 5432. Default on > > ASA connection timeout is 60 minutes. > > > > Webapps --> ASA Allow 25322 TCP/UDP --> Sequoia > > Sequoia --> ASA Allow 5432 TCP --> Postgresql > > > You are missing the ports required by the group communication > (depends on your configuration) for inter-controller communication. > You will also need to open the ports for the dump transfer operation > (backuper option). The connections between Sequoia are on the same layer 2 environment, and the controllers have been configured to allow the Multicast traffic defined by the appia.xml , controllers have no problem seeing each other. We store dump transfer on a NAS, and use nocopy option between the controllers, although the JMX ports are open to the controllers. > > > We get the following error messages > > > > ASA log > > May 16 12:33:17 XXXXX Deny TCP (no connection) from LAPP1/49838 to > > LDB1/5432 flags PSH ACK on interface XXX_XXX > > > Make sure that your connection pool timeout is lower than your ASA > connection timeout. > > cluster.log > > 2008-05-16 12:33:19,431 INFO Database backend XXXX.ldb1 is now in > > state disabled > > 2008-05-16 12:33:20,574 ERROR Disabling backend ldb1 because it is > > no more in sync > > > > full_cluster.log > > 2008-05-16 12:33:20,574 ERROR backend.DatabaseBackend.ldb1 Request > > 'update XXXXXXXXXXXXXXXXXX = ...' failed on backend ldb1 but 1 > > succeeded > > (org.continuent.sequoia.common.exceptions.BadConnectionException: > > An I/O error occured while sending to the backend.) > ASA broke the JDBC connection to PostgreSQL so the query execution > failed. You need to properly set your timeouts. These are our configs for the database backends. <VariablePoolConnectionManager initPoolSize="50" minPoolSize="20" maxPoolSize="300" idleTimeout="30" waitTimeout="10"/> >From what Robert said, it appears maybe I should have configured the MinPoolSize to 0? to close the connection completely when not needed. Is there any issue with having such a large initPoolSize, as we have multiple application servers accessing these, which generally grab the below amount. Is it possible that because the Minpoolsize is set to 20, that in fact the application wasn't creating enough data at those times, that 1 of the 20 were shutdown by the ASA as a result of enough inactivity? global.minSize = 50 global.poolSize = 200 > > As I am not an expert with JDBC connections, and how Sequoia uses > > the Postgresql JDBC connections to the backends, how can we specify > > it to use keepalives, or is this by default? I have read that > > Postgresql jdbc keepalive is off by default, referring to > > postgresql-jdbc-8.3-603 > > > > http://archives.postgresql.org/pgsql-jdbc/2008-02/msg00037.php > > > Sequoai just uses the standard JDBC API, so we have no control on > connections that are opened internally by the JDBC driver. > > Another interesting point, during the fail > > > > > > May 16 12:33:19 XXXXX : Deny icmp src XXXX:LAPP1 > > dst XXXXX:LDB1 (type 3, code 10) by access-group > > "XXXXXXX" [0x0, 0x0] > > > > > > It appears this is coming from our controller and sending to our > > database, the ASA is dropping this as only port 5432 TCP is > > allowed, no ICMP. What is it sending ICMP for, i realise this "port > > unreachable" code, is this a side affect of Sequoia loosing sync > > with the backend? > As far as I know there is no ICMP packet sent by the controller. This > might be a misconfiguration of the group communication that can use > PING packets to detect failures. I think this is to do with us not being able to access the port after we get an RST packet back from the ASA. The ASA gives us an RST packet when it closes any illegal port attempts or a communication it does not know about, and it appears there were RST packets in the logs. > > Thanks for your interest in Sequoia, > Emmanuel > As a result of the ASA dropping what it believes is inactive traffic after 1 hour, we have increased it to 24 hours and will monitor from there. As the ASA is an internal firewall behind our protected network, we have two options, set the minpoolsize=0 or turn off the connection timeout on the ASA and let the application decide when to close the connection. What do you guys suggest? We have been using Sequoia in test for almost a year and half, and a few weeks in production. We have recently migrated away from pgpoolv1. Thanks again, -- Stuart James _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
