Re: Problem running Apache Gump [vmgump-public]
On 14/09/2009, Stefan Bodewig bode...@apache.org wrote: On 2009-09-09, Stefan Bodewig bode...@apache.org wrote: I'm currently working going through the code in order to see where a retry on error logic could best be introduced. Looks as if it worked. Well done thanks! Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
RE: Problem running Apache Gump [vmgump-public]
-Original Message- From: Stefan Bodewig [mailto:bode...@apache.org] Sent: Sunday, September 13, 2009 9:00 PM To: general@gump.apache.org Subject: Re: Problem running Apache Gump [vmgump-public] On 2009-09-09, Stefan Bodewig bode...@apache.org wrote: I'm currently working going through the code in order to see where a retry on error logic could best be introduced. Looks as if it worked. Yes, cool Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
On 2009-09-09, Stefan Bodewig bode...@apache.org wrote: I'm currently working going through the code in order to see where a retry on error logic could best be introduced. Looks as if it worked. Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
- Original Message - From: Bill Barker billwbar...@verizon.net To: Gump code and data general@gump.apache.org Sent: Tuesday, September 08, 2009 7:11 PM Subject: Re: Problem running Apache Gump [vmgump-public] - Original Message - From: Stefan Bodewig bode...@apache.org To: general@gump.apache.org Sent: Tuesday, September 08, 2009 2:00 AM Subject: Re: Problem running Apache Gump [vmgump-public] On 2009-09-08, g...@vmgump.apache.org wrote: Failed to build project #[(704, 704)] : [test-ant], state:Failed SQL Warning:class '_mysql_exceptions.OperationalError':(2006, 'MySQL server has gone away') We keep getting this since a few weeks over and over again. I guess it started when somebody upgraded vmgump's OS (many thanks for that!) and either a newer MySQL or python-mysqldb isn't working as expected. What I can tell by now is that the database is still there (executed a few minutes after the emial arrived): It has a correlation of nearly 1 with commons-lang-2.x failing. I'm guessing that since the later event hoses all M2 builds, that Gump is temporarily running out of connections with so many projects failing quickly. It's only a theory, but it is the one that I based splitting the tests out of commons-lang-2.x. Ok, so much for that theory :(. , | r...@vmgump:/var/log# /etc/init.d/mysql status | * /usr/bin/mysqladmin Ver 8.41 Distrib 5.0.51a, for debian-linux-gnu on i486 | Copyright (C) 2000-2006 MySQL AB | This software comes with ABSOLUTELY NO WARRANTY. This is free software, | and you are welcome to modify and redistribute it under the GPL license | | Server version 5.0.51a-3ubuntu5.4 | Protocol version10 | Connection Localhost via UNIX socket | UNIX socket /var/run/mysqld/mysqld.sock | Uptime: 1 day 23 hours 11 min 10 sec | | Threads: 1 Questions: 6286 Slow queries: 0 Opens: 34 Flush tables: 1 Open tables: 28 Queries per second avg: 0.037 ` and hasn't restarted during the Gump run. http://dev.mysql.com/doc/refman/5.0/en/gone-away.html hints at various reasons why this might happen. The server side timeout is at eight hours and we shouldn't be hitting that. It seems as if the python wrapper was not enabling automatic reconnect and unfortunately the python-mysqldb version installed by hardy (1.2.2) doesn't have the ping(reconnect) method later versions have added (by looking through python-mysqldb's svn logs, that is), mysql error logs don't say anything (or I just don't find them, quite possible). My next step would be to wrap each execute() going against the database in a try/catch that clears the connection and creates a new one if an OperationalError is caught. But before I get my feet wet with another area of Python I've never used, maybe anybody knows of a better approach. BTW, after this type of error occurs we get Gump runs where bcel and many other early building mvn projects fail. This is because the mvn proxy of the previous run is still up and thinks it could serve commons-lang but the jar has already been removed by the sync operation (and thus mvn fails because it cannot download commons-lang). This needs to be fixed in a separate way, either by making sure the mvn proxy is shut down at the end or by clearing it at the start (the later is probably easier to implement). Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
On 2009-09-09, Bill Barker billwbar...@verizon.net wrote: Ok, so much for that theory :(. Those others really fail because the mvn proxy is returning a 404 on the commons-lang jar because it thinks it can serve it but the jar is no longer there. It looks as if the problem occurs when the complete run reaches eight hours or more. Then again I'd expect the statistics actor to use the connection after each and every build so the connection is supposed to be busy - but I'm not yet familiar with that part of the Gump code base and have zero experience with mysql myself (we prefer to pay for our RDBMS 8-). I'm currently working going through the code in order to see where a retry on error logic could best be introduced. Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
On 2009-09-08, g...@vmgump.apache.org wrote: Failed to build project #[(704, 704)] : [test-ant], state:Failed SQL Warning:class '_mysql_exceptions.OperationalError':(2006, 'MySQL server has gone away') We keep getting this since a few weeks over and over again. I guess it started when somebody upgraded vmgump's OS (many thanks for that!) and either a newer MySQL or python-mysqldb isn't working as expected. What I can tell by now is that the database is still there (executed a few minutes after the emial arrived): , | r...@vmgump:/var/log# /etc/init.d/mysql status | * /usr/bin/mysqladmin Ver 8.41 Distrib 5.0.51a, for debian-linux-gnu on i486 | Copyright (C) 2000-2006 MySQL AB | This software comes with ABSOLUTELY NO WARRANTY. This is free software, | and you are welcome to modify and redistribute it under the GPL license | | Server version 5.0.51a-3ubuntu5.4 | Protocol version10 | Connection Localhost via UNIX socket | UNIX socket /var/run/mysqld/mysqld.sock | Uptime: 1 day 23 hours 11 min 10 sec | | Threads: 1 Questions: 6286 Slow queries: 0 Opens: 34 Flush tables: 1 Open tables: 28 Queries per second avg: 0.037 ` and hasn't restarted during the Gump run. http://dev.mysql.com/doc/refman/5.0/en/gone-away.html hints at various reasons why this might happen. The server side timeout is at eight hours and we shouldn't be hitting that. It seems as if the python wrapper was not enabling automatic reconnect and unfortunately the python-mysqldb version installed by hardy (1.2.2) doesn't have the ping(reconnect) method later versions have added (by looking through python-mysqldb's svn logs, that is), mysql error logs don't say anything (or I just don't find them, quite possible). My next step would be to wrap each execute() going against the database in a try/catch that clears the connection and creates a new one if an OperationalError is caught. But before I get my feet wet with another area of Python I've never used, maybe anybody knows of a better approach. BTW, after this type of error occurs we get Gump runs where bcel and many other early building mvn projects fail. This is because the mvn proxy of the previous run is still up and thinks it could serve commons-lang but the jar has already been removed by the sync operation (and thus mvn fails because it cannot download commons-lang). This needs to be fixed in a separate way, either by making sure the mvn proxy is shut down at the end or by clearing it at the start (the later is probably easier to implement). Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
- Original Message - From: Stefan Bodewig bode...@apache.org To: general@gump.apache.org Sent: Tuesday, September 08, 2009 2:00 AM Subject: Re: Problem running Apache Gump [vmgump-public] On 2009-09-08, g...@vmgump.apache.org wrote: Failed to build project #[(704, 704)] : [test-ant], state:Failed SQL Warning:class '_mysql_exceptions.OperationalError':(2006, 'MySQL server has gone away') We keep getting this since a few weeks over and over again. I guess it started when somebody upgraded vmgump's OS (many thanks for that!) and either a newer MySQL or python-mysqldb isn't working as expected. What I can tell by now is that the database is still there (executed a few minutes after the emial arrived): It has a correlation of nearly 1 with commons-lang-2.x failing. I'm guessing that since the later event hoses all M2 builds, that Gump is temporarily running out of connections with so many projects failing quickly. It's only a theory, but it is the one that I based splitting the tests out of commons-lang-2.x. , | r...@vmgump:/var/log# /etc/init.d/mysql status | * /usr/bin/mysqladmin Ver 8.41 Distrib 5.0.51a, for debian-linux-gnu on i486 | Copyright (C) 2000-2006 MySQL AB | This software comes with ABSOLUTELY NO WARRANTY. This is free software, | and you are welcome to modify and redistribute it under the GPL license | | Server version 5.0.51a-3ubuntu5.4 | Protocol version10 | Connection Localhost via UNIX socket | UNIX socket /var/run/mysqld/mysqld.sock | Uptime: 1 day 23 hours 11 min 10 sec | | Threads: 1 Questions: 6286 Slow queries: 0 Opens: 34 Flush tables: 1 Open tables: 28 Queries per second avg: 0.037 ` and hasn't restarted during the Gump run. http://dev.mysql.com/doc/refman/5.0/en/gone-away.html hints at various reasons why this might happen. The server side timeout is at eight hours and we shouldn't be hitting that. It seems as if the python wrapper was not enabling automatic reconnect and unfortunately the python-mysqldb version installed by hardy (1.2.2) doesn't have the ping(reconnect) method later versions have added (by looking through python-mysqldb's svn logs, that is), mysql error logs don't say anything (or I just don't find them, quite possible). My next step would be to wrap each execute() going against the database in a try/catch that clears the connection and creates a new one if an OperationalError is caught. But before I get my feet wet with another area of Python I've never used, maybe anybody knows of a better approach. BTW, after this type of error occurs we get Gump runs where bcel and many other early building mvn projects fail. This is because the mvn proxy of the previous run is still up and thinks it could serve commons-lang but the jar has already been removed by the sync operation (and thus mvn fails because it cannot download commons-lang). This needs to be fixed in a separate way, either by making sure the mvn proxy is shut down at the end or by clearing it at the start (the later is probably easier to implement). Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Problem running Apache Gump [vmgump-public]
this was me.
On Jul 8, 2007, at 2:55 AM, [EMAIL PROTECTED] wrote:
There is a problem with run 'vmgump-public' (07072007_160002),
location : http://vmgump.apache.org/gump/public
The log ought be at:
http://vmgump.apache.org/gump/public/gump_log.txt
The last (up to) 50 lines of the log are :
-- Extracted (fallback) artifacts from Repository : jakarta-
turbine-2
Perform Update on #[(291, 291)] : portals-jetspeed-1
Perform SVN Update on #[(291, 291)] : portals-jetspeed-1
Build Project: #[(696, 696)] : portals-jetspeed-1 :
[state:Prerequisite Failed]
SQL Warning:_mysql_exceptions.OperationalError:(2002, Can't
connect to local MySQL server through socket '/var/run/mysqld/
mysqld.sock' (2))
SQL Warning:_mysql_exceptions.OperationalError:(2002, Can't
connect to local MySQL server through socket '/var/run/mysqld/
mysqld.sock' (2))
SQL Error on [SELECT * FROM gump_public.gump_workspace_stats WHERE
workspace_name='vmgump-public'] : (2002, Can't connect to local
MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))
Traceback (most recent call last):
File /x1/gump/public/gump/python/gump/util/mysql.py, line 145,
in select
affected=cursor.execute(statement)
File /var/lib/python-support/python2.4/MySQLdb/cursors.py, line
166, in execute
self.errorhandler(self, exc, value)
File /var/lib/python-support/python2.4/MySQLdb/connections.py,
line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (2002, Can't connect to local MySQL server
through socket '/var/run/mysqld/mysqld.sock' (2))
Traceback (most recent call last):
File bin/integrate.py, line 114, in ?
irun()
File bin/integrate.py, line 91, in irun
result = getRunner(run).perform()
File /x1/gump/public/gump/python/gump/core/runner/runner.py,
line 254, in perform
return self.performRun()
File /x1/gump/public/gump/python/gump/core/runner/demand.py,
line 215, in performRun
self.finalize()
File /x1/gump/public/gump/python/gump/core/runner/runner.py,
line 232, in finalize
self.run._dispatchEvent(FinalizeRunEvent(self.run))
File /x1/gump/public/gump/python/gump/core/run/gumprun.py, line
186, in _dispatchEvent
actor._processEvent(event)
File /x1/gump/public/gump/python/gump/core/run/actor.py, line
83, in _processEvent
self.processEvent(event)
File /x1/gump/public/gump/python/gump/core/run/actor.py, line
127, in processEvent
self._processOtherEvent(event)
File /x1/gump/public/gump/python/gump/core/run/actor.py, line
184, in _processOtherEvent
self.processOtherEvent(event)
File /x1/gump/public/gump/python/gump/actor/stats/
statistician.py, line 66, in processOtherEvent
self.updateStatistics()
File /x1/gump/public/gump/python/gump/actor/stats/
statistician.py, line 113, in updateStatistics
ws=self.db.getWorkspaceStats(self.workspace.getName())
File /x1/gump/public/gump/python/gump/actor/stats/mysql/
statsdb.py, line 87, in getWorkspaceStats
self._getStats
('gump_workspace_stats','workspace_name',workspaceName,stats)
File /x1/gump/public/gump/python/gump/actor/stats/mysql/
statsdb.py, line 163, in _getStats
StatisticsDB.ATTR_COLUMN_MAP.values())
File /x1/gump/public/gump/python/gump/util/mysql.py, line 145,
in select
affected=cursor.execute(statement)
File /var/lib/python-support/python2.4/MySQLdb/cursors.py, line
166, in execute
self.errorhandler(self, exc, value)
File /var/lib/python-support/python2.4/MySQLdb/connections.py,
line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (2002, Can't connect to local
MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))
Process Exit Code : 1
--
Gump Version: 2.0.2-alpha-0003
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
regards,
Leo Simons
--
[EMAIL PROTECTED] / +31616471562
Joost Technologies B.V.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problem running Apache Gump [vmgump-public]
On Nov 14, 2006, at 4:08 PM, [EMAIL PROTECTED] wrote: I would say there's a problem! And start looking at the time zone settings of this VM (or its underlying infrastructure). S. -- [EMAIL PROTECTED]http://www.temme.net/sander/ PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problem running Apache Gump [vmgump-public]
On Tue, 14 Nov 2006, Sander Temme [EMAIL PROTECTED] wrote: On Nov 14, 2006, at 4:08 PM, [EMAIL PROTECTED] wrote: I would say there's a problem! True. The smartfrog project descriptor said it was using a svn repository while the repository descriptor was a CVS one. I fixed it, at least I think so. You'll need to remove the working copy of smartfrog once again, though. And start looking at the time zone settings of this VM (or its underlying infrastructure). $ date Tue Nov 14 21:22:05 PST 2006 at 5:22 GMT, looks OK, doesn't it? Stefan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problem running Apache Gump [vmgump-public]
Gang, In these notifications, the ASF spam check (whatever that is) injects a header; X-ASF-Spam-Status: No, hits=-9.6 required=10.0 tests=ALL_TRUSTED,INVALID_DATE,NO_REAL_NAME And I personally, check if INVALID_DATE is there and send it to spambox. Gumps date format; Date: 19 Jul 2005 05:26:38 Known working date format; Date: Mon, 18 Jul 2005 17:16:25 +0200 I suspect it is a missing timezone that triggers it, but I don't know RFC822 (I think) by heart, and the format could be fairly strict. Anyone interested in fixing such a minute issue ?? And I ain't going to learn Python, just for that :o) Cheers Niclas On Tuesday 19 July 2005 13:26, [EMAIL PROTECTED] wrote: There is a problem with run 'vmgump-public' (17072005_180001), location : http://vmgump.apache.org/gump/public The log ought be at: http://vmgump.apache.org/gump/public/gump_log.txt The last (up to) 50 lines of the log are : -- Perform Artifact Repository Search for : cocoon-block-slide - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problem running Apache Gump [vmgump-public]
after this one, Gump didn't clean up the locks it held. I've manually removed the lock files and killed about five python processes, this should have been enough to get the next official run (ten minutes from now) started. out.txt doesn't hold any information that hasn't been part of this mail, at least not for me. I've saved out.txt and the log directory to ~gump/20050616 on vmgump. Stefan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
