SRU instructions in the bug description above. ** Description changed:
Using 1.6.2~bzr1120-0ubuntu1, I can't get the credentials: ubu...@uec-cc:~$ sudo euca_conf --get-credentials mycreds.zip ERROR: you need to be on the CLC host and the CLC needs to be running. A wget on the register url also times out: ubu...@uec-cc:~$ wget -T 10 -t 1 -O - --no-check-certificate https://127.0.0.1:8443/register --2010-01-04 20:39:34-- https://127.0.0.1:8443/register Connecting to 127.0.0.1:8443... connected. WARNING: cannot verify 127.0.0.1's certificate, issued by `/C=US/ST=CA/L=Santa Barbara/O=db/OU=Eucalyptus/CN=localhost': - Self-signed certificate encountered. + Self-signed certificate encountered. WARNING: certificate common name `localhost' doesn't match requested host name `127.0.0.1'. HTTP request sent, awaiting response... Read error (Connection timed out) in headers. Giving up. The eucalyptus-cloud process is running a listening on port 8443. I can see the following errors in /var/log/eucalyptus/cloud-error.log: 0:36:55 [log:653891...@qtp-1693378617-9] ERROR /register java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection - at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46) - at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98) - at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91) - at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117) - at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199) - at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210) - at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) - at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) - at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) - at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389) - at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) - at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) - at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) - at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) - at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) - at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) - at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) - at org.mortbay.jetty.Server.handle(Server.java:326) - at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) - at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) - at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539) - at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) - at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) - at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) - at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) + at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46) + at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98) + at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91) + at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117) + at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199) + at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210) + at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) + at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) + at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) + at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389) + at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) + at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) + at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) + at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) + at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) + at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) + at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) + at org.mortbay.jetty.Server.handle(Server.java:326) + at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) + at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) + at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539) + at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) + at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) + at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) + at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection - at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614) - at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41) - at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40) - ... 24 more + at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614) + at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41) + at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40) + ... 24 more Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection - at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97) - at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) - at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) - at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) - at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) - at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142) - at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85) - at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353) - at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38) - ... 25 more + at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97) + at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) + at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) + at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) + at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) + at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142) + at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85) + at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353) + at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38) + ... 25 more Caused by: java.sql.SQLException: Connection is broken: java.net.SocketException: Connection timed out - at org.hsqldb.jdbc.Util.sqlException(Unknown Source) - at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source) - at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) - at java.lang.reflect.Method.invoke(Method.java:616) - at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162) - at $Proxy27.getAutoCommit(Unknown Source) - at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81) - at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) - ... 30 more + at org.hsqldb.jdbc.Util.sqlException(Unknown Source) + at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source) + at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:616) + at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162) + at $Proxy27.getAutoCommit(Unknown Source) + at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81) + at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) + ... 30 more + + == SRU == + + IMPACT: Users of 9.10 UEC will often experience long, non-deterministic + delays (10-20 minutes in many cases) when restarting eucalyptus services + (which includes reboots, package upgrades, service restarts). This is + highly inconvenient, yielding UEC unusable until database network + connections reset. + + HOW FIXED: The fix consists of two trivial iptables calls being added to + the eucalyptus upstart script. Upstream had these calls in there init + scripts, but were inadvertently dropped when porting Eucalyptus to + upstart. These iptables commands will ensure that the iptables kernel + module (and most importantly, the ip connection tracker) is loaded and + active before Eucalyptus comes up. WIthout said ip connection tracker, + Eucalyptus will often establish a connection to the database, then + iptables is loaded and connections are mangled, breaking the connection + to the database. The user will see the problem in any one of a number + of disguising ways (front end not working, api tools not responding, + etc). All of these problems are due to an inaccessible database. After + a while (10-20 minutes), Eucalyptus will reset the database connection. + With this fix, the above problems should never happen. Eucalyptus + should be back up and running within 1-2 minutes of boot (if not + immediately). + + MINIMAL PATCH: + diff -u eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart + --- eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart + +++ eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart + @@ -11,6 +11,10 @@ + # Check if installed + [ -f /usr/sbin/euca_conf ] || { stop; exit 0; } + + + # Ensure that the iptables module gets loaded here + + iptables -t nat -L -n >/dev/null + + iptables -L -n > /dev/null + + + mkdir -p /var/run/eucalyptus/net + chown eucalyptus:eucalyptus /var/run/eucalyptus /var/run/eucalyptus/net + + + REPRODUCING THE BUG: + Reboot your UEC (or sudo restart eucalyptus). If restarting eucalyptus takes a *long* time, you are experiencing one symptom of this bug. Once upstart thinks that eucalyptus is up, try: $(sudo wget --no-check-certificate https://localhost:8443) If this takes a long time, or fails to work, you are experiencing a symptom of this bug. Note that the problem is inherently due to a race condition, and therefore may not be immediately reproducible. Try rebooting/restarting a few times, and you will likely hit it. + + REGRESSION POTENTIAL: + I cannot see any possible regression potential. The iptables modules will be loaded eventually. This patch just ensures that they get loaded before Eucalyptus tries to start services. -- [SRU] eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
