Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
hi teodor, thanks for looking into this! On Wed, Sep 22, 2010 at 02:58:28AM +0300, Teodor MICU wrote: I've been able to reproduce the problem. So far the problem will appear any time I'll activate 'spine' poller. One weird thing is that both CMDPHP and SPINE are present in the cacti.log as if both are running even if I selected 'spine' to do the polling. Note that I'm using cacti=0.8.7g with spine=0.8.7a-2.3 on a Debian 5.0 (lenny) host. do you get the same problems if you also upgrade spine from the lenny version to the squeeze version (0.8.7e-2)? i wouldn't be surprised that the poller from lenny is designed for different sql/schema and thus might not do the right INSERT/UPDATE type statements. if upgrading to the squeeze version of spine doesn't solve the problem, you could also try updating to the latest upstream version (i.e. compiling it yourself and replacing the packaged version), but i haven't been lead to believe that the latest version is so different from the squeeze/sid version. sean signature.asc Description: Digital signature
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
Hi,
On Tue, Sep 21, 2010 at 11:56 PM, sean finney sean...@debian.org wrote:
some ideas/questions off the top of my head:
* did you get any sql upgrade errors after the package was updated?
There were no sql upgrade scripts from 0.8.7e-4 to 0.8.7g-1. The
upgrade says clearly that there are no scripts to run. I've checked
manually in the install directory and these two scripts are just
changing the cacti vesion in the database:
- 0_8_7e_to_0_8_7f.php
- 0_8_7f_to_0_8_7g.php
* does the schema seem correct (i.e. a disabled column in the host table)?
I don't know anything related to a 'schema', so probably not the
issue. If you can tell me what to check, I'll do.
* does it work if you disable the spine poller and use the cmd.php one
instead?
Yes, with 'cmd.php' I get the graphs but still at every 5 minutes
there are SQL errors reported (see attached).
No, with 'spine' I still get the SQL errors but also only NaN on all graphs.
* does it start graphing again if you nuke the rrd files or play around
with rrdtool versions?
I'll try to avoid playing around with rrdtool again. I did it for the
conversion from 1.2x to 1.3x and this is the main reason I'm using
cacti from squeeze on a lenny host.
esp for the first two items, if you suspect the problem is in the SQL,
and have time to help with some QA: i think it would be very helpful to
get a db dump of a fresh install vs the db dump of a almost-as-fresh
install of the previous version upgraded after maybe half an hour of
polling... it could be that there's a bug in the update sql logic.
I'll try to do a fresh install, but I don't I'll have the time this week.
Thanks
09/22/2010 10:55:34 AM - PHPSVR: Poller[0] Maximum runtime of 60 seconds exceeded for the Script Server. Exiting.
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_recache','RecacheTime:0.0 HostsRecached:0')'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT COUNT(*) FROM poller_command
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_poller','Time:0.0022 Method:cmd.php Processes:1 Threads:N/A Hosts:0 HostsPerProcess:0 DataSources:456 RRDsProcessed:0')'
09/22/2010 10:55:33 AM - SYSTEM STATS: Time:0.0022 Method:cmd.php Processes:1 Threads:N/A Hosts:0 HostsPerProcess:0 DataSources:456 RRDsProcessed:0
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE poller_id=0 AND end_time'-00-00 00:00:00'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('date',NOW())'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT local_data_id, rrd_name FROM poller_output
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:TRUNCATE TABLE poller_time'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE end_time='-00-00 00:00:00'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('path_webroot','/usr/share/cacti/site')'
09/22/2010 10:55:33 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT id FROM host WHERE disabled = '' ORDER BY id
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
Hi, On Wed, Sep 22, 2010 at 9:23 AM, sean finney sean...@debian.org wrote: do you get the same problems if you also upgrade spine from the lenny version to the squeeze version (0.8.7e-2)? i wouldn't be surprised that the poller from lenny is designed for different sql/schema and thus might not do the right INSERT/UPDATE type statements. I didn't upgrade spine because I'll have to upgrade mysql too (and probably other packages from backports). The weird thing about these SQL errors is that running them manually gives no error but sometimes there are no results returned from the query. if upgrading to the squeeze version of spine doesn't solve the problem, you could also try updating to the latest upstream version (i.e. compiling it yourself and replacing the packaged version), but i haven't been lead to believe that the latest version is so different from the squeeze/sid version. There are near zero chances of compiling it myself. I'll probably try a fresh install on a squeeze host with cacti and spine =0.8.7e. Until them I'll stick with cacti=0.8.7e-4. Thanks -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
hiya, i'll group two replies together here. On Wed, Sep 22, 2010 at 10:57:10AM +0300, Teodor MICU wrote: * does the schema seem correct (i.e. a disabled column in the host table)? I don't know anything related to a 'schema', so probably not the issue. If you can tell me what to check, I'll do. what i meant was something along the lines of comparing mysqldump output when the --no-data option was used (i.e. are all the tables/columns the same between the versions and do the queries look correct). * does it work if you disable the spine poller and use the cmd.php one instead? Yes, with 'cmd.php' I get the graphs but still at every 5 minutes there are SQL errors reported (see attached). No, with 'spine' I still get the SQL errors but also only NaN on all graphs. ok, then my suspicions turn towards spine as the culprit. * does it start graphing again if you nuke the rrd files or play around with rrdtool versions? I'll try to avoid playing around with rrdtool again. I did it for the conversion from 1.2x to 1.3x and this is the main reason I'm using cacti from squeeze on a lenny host. right. and if cmd.php works and spine doesn't, i don't think we should focus too hard on the rrdtool goo. On Wed, Sep 22, 2010 at 11:02:05AM +0300, Teodor MICU wrote: do you get the same problems if you also upgrade spine from the lenny version to the squeeze version (0.8.7e-2)? i wouldn't be surprised that the poller from lenny is designed for different sql/schema and thus might not do the right INSERT/UPDATE type statements. I didn't upgrade spine because I'll have to upgrade mysql too (and probably other packages from backports). The weird thing about these SQL errors is that running them manually gives no error but sometimes there are no results returned from the query. okay, this may be where the problem is, though. if upgrading to the squeeze version of spine doesn't solve the problem, you could also try updating to the latest upstream version (i.e. compiling it yourself and replacing the packaged version), but i haven't been lead to believe that the latest version is so different from the squeeze/sid version. There are near zero chances of compiling it myself. I'll probably try a fresh install on a squeeze host with cacti and spine =0.8.7e. Until them I'll stick with cacti=0.8.7e-4. i think backporting the package should be fairly straightforward, assuming your problem is because you don't want to get your hands dirty at least (vs some kinda corporate you can not compile on the web hosts thing). the following should be all you need to do: * make sure there is a deb-src line for squeeze on your lenny box deb-src http://ftp.se.debian.org/debian squeeze main * (as root) apt-get update; apt-get install build-essential fakeroot * (as root) apt-get build-dep cacti-spine * apt-get source -b cacti packages compiled against your lenny packages (i.e. no mysql upgrade required) will then have appeared in the current working directory. i suggest you give it a go before getting to deep into any other troubleshooting. as it should only take a number of minutes to do the above and then a couple runs of spine after that to see if it works. sean -- signature.asc Description: Digital signature
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
retitle 531546 empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
quit
Hi,
It seems that this old problem has an impact on 'squeeze' too. After
upgrading from v0.8.7e-4 to 0.8.7g-1 all the graphs were with NaN from
this point forward. The error message in cacti.log was like in the
attached file. I've downgraded to the previous version and restored
the database from a backup (cacti refused to continue due to a higher
version 7g than 7e currently installed).
I'll probably attempt a new upgrade soon, do you have any recommendations?
Thanks
09/21/2010 08:00:39 PM - SYSTEM STATS: Time:37.3214 Method:spine Processes:2 Threads:8 Hosts:42 HostsPerProcess:21 DataSources:464 RRDsProcessed:411
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT id FROM host WHERE disabled = '' ORDER BY id
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('path_webroot','/usr/share/cacti/site')'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE end_time='-00-00 00:00:00'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:TRUNCATE TABLE poller_time'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT local_data_id, rrd_name FROM poller_output
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('date',NOW())'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE poller_id=0 AND end_time'-00-00 00:00:00'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)
09/21/2010 08:01:01 PM - SYSTEM STATS: Time:0.0020 Method:spine Processes:2 Threads:8 Hosts:0 HostsPerProcess:0 DataSources:464 RRDsProcessed:0
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_poller','Time:0.0020 Method:spine Processes:2 Threads:8 Hosts:0 HostsPerProcess:0 DataSources:464 RRDsProcessed:0')'
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT COUNT(*) FROM poller_command
09/21/2010 08:01:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_recache','RecacheTime:0.0 HostsRecached:0')'
09/21/2010 08:02:01 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT id FROM host WHERE disabled = '' ORDER BY id
09/21/2010 08:02:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('path_webroot','/usr/share/cacti/site')'
09/21/2010 08:02:01 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE end_time='-00-00 00:00:00'
09/21/2010 08:02:01 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:TRUNCATE TABLE poller_time'
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
hi, On Tue, Sep 21, 2010 at 11:02:37PM +0300, Teodor MICU wrote: I'll probably attempt a new upgrade soon, do you have any recommendations? some ideas/questions off the top of my head: * did you get any sql upgrade errors after the package was updated? * does the schema seem correct (i.e. a disabled column in the host table)? * does it work if you disable the spine poller and use the cmd.php one instead? * does it start graphing again if you nuke the rrd files or play around with rrdtool versions? esp for the first two items, if you suspect the problem is in the SQL, and have time to help with some QA: i think it would be very helpful to get a db dump of a fresh install vs the db dump of a almost-as-fresh install of the previous version upgraded after maybe half an hour of polling... it could be that there's a bug in the update sql logic. sean signature.asc Description: Digital signature
Bug#531546: Bug #531546: empty graphs due to 'A DB Exec Failed' Poller[0] ERROR
Hi again,
On Tue, Sep 21, 2010 at 11:02 PM, Teodor MICU mteo...@gmail.com wrote:
It seems that this old problem has an impact on 'squeeze' too. After
upgrading from v0.8.7e-4 to 0.8.7g-1 all the graphs were with NaN from
this point forward.
I've been able to reproduce the problem. So far the problem will
appear any time I'll activate 'spine' poller. One weird thing is that
both CMDPHP and SPINE are present in the cacti.log as if both are
running even if I selected 'spine' to do the polling. Note that I'm
using cacti=0.8.7g with spine=0.8.7a-2.3 on a Debian 5.0 (lenny) host.
So, using 'cmd.php' is an option but brings some important drawbacks:
- there are many U results from the queries, with 'spine' there were
not so many;
- it doesn't have enough time to do all the polling (for all
graphs/data sources).
Update: using 'cmd.php' also gives SQL errors, although it could be
from another polling process. I suspect that 'spine' is causing some
problems to the DB that affects 'cmd.php' too.
Thanks
09/21/2010 11:30:01 PM - POLLER: Poller[0] WARNING: There are '2' detected as overrunning a polling process, please investigate
09/21/2010 11:30:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 3, Data Sources: traffic_in(DS[11]), traffic_out(DS[11]), cpu(DS[365])
09/21/2010 11:30:01 PM - SPINE: Poller[0] Host[3] DS[33] WARNING: Result from SNMP not valid. Partial Result: ...
09/21/2010 11:30:01 PM - SPINE: Poller[0] Host[3] DS[33] WARNING: Result from SNMP not valid. Partial Result: ...
09/21/2010 11:30:01 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
09/21/2010 11:30:01 PM - SYSTEM STATS: Time:59.3551 Method:cmd.php Processes:2 Threads:N/A Hosts:42 HostsPerProcess:21 DataSources:464 RRDsProcessed:301
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT id FROM host WHERE disabled = '' ORDER BY id
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('path_webroot','/usr/share/cacti/site')'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE end_time='-00-00 00:00:00'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:TRUNCATE TABLE poller_time'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT local_data_id, rrd_name FROM poller_output
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('date',NOW())'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT count(*) FROM poller_time WHERE poller_id=0 AND end_time'-00-00 00:00:00'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)
09/21/2010 11:30:02 PM - SYSTEM STATS: Time:0.0022 Method:cmd.php Processes:2 Threads:N/A Hosts:0 HostsPerProcess:0 DataSources:464 RRDsProcessed:0
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_poller','Time:0.0022 Method:cmd.php Processes:2 Threads:N/A Hosts:0 HostsPerProcess:0 DataSources:464 RRDsProcessed:0')'
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'0', SQL:SELECT COUNT(*) FROM poller_command
09/21/2010 11:30:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'0', SQL:REPLACE INTO settings (name,value) VALUES ('stats_recache','RecacheTime:0.0 HostsRecached:0')'
09/21/2010 11:30:03 PM - PHPSVR: Poller[0] Maximum runtime of 60 seconds exceeded for the Script Server. Exiting.
09/22/2010 02:49:43 AM - SYSTEM STATS: Time:96.2642 Method:cmd.php Processes:1 Threads:N/A Hosts:42 HostsPerProcess:42 DataSources:466 RRDsProcessed:412
09/22/2010 02:50:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
09/22/2010 02:50:00 AM - SYSTEM STATS: Time:17.1356 Method:cmd.php Processes:1 Threads:N/A Hosts:42 HostsPerProcess:42 DataSources:466 RRDsProcessed:45
09/22/2010 02:50:02 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
09/22/2010 02:50:02 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: cpu(DS[308])
09/22/2010 02:50:06 AM - CMDPHP: Poller[0] Host[3] DS[32] WARNING: Result from SNMP not valid. Partial Result: U
09/22/2010 02:50:06 AM - CMDPHP: Poller[0] Host[3] DS[32] WARNING: Result from SNMP not valid. Partial Result: U
09/22/2010 02:50:43 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:SELECT id FROM host WHERE disabled =
