[rt-users] AssetTracker crashes loading asset...raw horsepower solution?
On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution?
How many assets do you have? Nobody has ever seen this issue before On Tue, Nov 11, 2008 at 12:57 PM, John [EMAIL PROTECTED] wrote: On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http
Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution?
9,359 On Tue, 11 Nov 2008, Todd Chapman wrote: Date: Tue, 11 Nov 2008 13:04:14 -0500 From: Todd Chapman [EMAIL PROTECTED] To: John [EMAIL PROTECTED] Cc: Curtis Bruneau [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution? How many assets do you have? Nobody has ever seen this issue before On Tue, Nov 11, 2008 at 12:57 PM, John [EMAIL PROTECTED] wrote: On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets
[rt-users] AssetTracker crashes loading asset
well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
not certain it makes much sense...ive never had this problem in rt 3.4.5 with at 1.2.3.. On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
While my issue is not related to asset tracker, the behavior sounds identical. It seems to be related to how RT handles search and display, the only thing I can think of is the menu on the left, It has to determine the first and last record and also next and prev, I think it's doing a full scan to figure it out.. pretty intensive operation, It would be better if it didn't have to deal with table.* and say only table.ticketid. First Prev #12345 Next Last Sorry for the confusion, could just be a coincidence. Curtis John wrote: not certain it makes much sense...ive never had this problem in rt 3.4.5 with at 1.2.3.. On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
hrm...i just pulled in a list of 47,000 RT tickets and was able to load a single ticket after a time of about 20 seconds...seemed normal. have you tried Set($LogToSyslog , 'alert'); as a solution? this seems isolated to assettracker...i think... On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 13:12:53 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset While my issue is not related to asset tracker, the behavior sounds identical. It seems to be related to how RT handles search and display, the only thing I can think of is the menu on the left, It has to determine the first and last record and also next and prev, I think it's doing a full scan to figure it out.. pretty intensive operation, It would be better if it didn't have to deal with table.* and say only table.ticketid. First Prev #12345 Next Last Sorry for the confusion, could just be a coincidence. Curtis John wrote: not certain it makes much sense...ive never had this problem in rt 3.4.5 with at 1.2.3.. On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset
That's around near the limit of what I can pull in. While you are probably right with it being isolated, 20 seconds does seem a little suspect. I'm curious how much memory is being used during that call. We have a lot of attachments which makes each record rather large at times. If you do end up finding a solution to your problem I'm curious to hear it. John wrote: hrm...i just pulled in a list of 47,000 RT tickets and was able to load a single ticket after a time of about 20 seconds...seemed normal. have you tried Set($LogToSyslog , 'alert'); as a solution? this seems isolated to assettracker...i think... On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 13:12:53 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset While my issue is not related to asset tracker, the behavior sounds identical. It seems to be related to how RT handles search and display, the only thing I can think of is the menu on the left, It has to determine the first and last record and also next and prev, I think it's doing a full scan to figure it out.. pretty intensive operation, It would be better if it didn't have to deal with table.* and say only table.ticketid. First Prev #12345 Next Last Sorry for the confusion, could just be a coincidence. Curtis John wrote: not certain it makes much sense...ive never had this problem in rt 3.4.5 with at 1.2.3.. On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com
