Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
It seems to occur for any ticket. I tried it on our most innocuous queue and it still occurred. Removed all people from the ticket, no special things in the ticket, and it still causes its apache process to lock up the system. Unfortunately I don't currently have the resources to set up a test RT system to see if it occurs. I'm running Ubuntu 12.04.4, perl 5.14.2, Apache 2.2.22, and MySQL 5.5.35-0ubuntu0.12.04.1-log on Ubuntu 12.04.4 -Original Message- From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Alex Vandiver Sent: Monday, January 27, 2014 2:19 PM To: rt-users@lists.bestpractical.com Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2 On Fri, 2014-01-24 at 17:43 +, Jonah Hirsch wrote: I just upgraded to RT 4.2.2 yesterday, and whenever a user merges a ticket, the responsible apache processes maxes out memory usage on the system and the system slows to a crawl until the OS kills the process, since the system is out of memory. Is this a known issue? Anyone have any thoughts on what I can do to try to solve this problem? This is the first I've heard of it. Does it occur for any ticket merge? Can you replicate it in a clean RT 4.2.2? - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
On Tue, 2014-01-28 at 16:23 +, Jonah Hirsch wrote: It seems to occur for any ticket. I tried it on our most innocuous queue and it still occurred. Removed all people from the ticket, no special things in the ticket, and it still causes its apache process to lock up the system. Do you have any extensions or customizations installed? Are there any errors in your error logs? - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
I get nothing in logs with debugging on before the system locks up. The issue still occurs after disabling all extensions but RT::Authen::ExternalAuth. -Original Message- From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Alex Vandiver Sent: Tuesday, January 28, 2014 11:45 AM To: rt-users@lists.bestpractical.com Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2 On Tue, 2014-01-28 at 16:23 +, Jonah Hirsch wrote: It seems to occur for any ticket. I tried it on our most innocuous queue and it still occurred. Removed all people from the ticket, no special things in the ticket, and it still causes its apache process to lock up the system. Do you have any extensions or customizations installed? Are there any errors in your error logs? - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
I just remembered that when upgrading I had some database upgrade errors. I checked the upgrade history in RT and saw that it failed upgrading the DB schema from 4.1.12 on. When I try to run /opt/rt4/sbin/rt-setup-database --action insert --datafile content From the /rt-4.2.2/etc/upgrade/4.1.12 directory I get the following including an error: Now inserting data. [52450] [Tue Jan 28 19:16:07 2014] [debug]: Going to load 'content' data file (/opt/rt4/sbin/../lib/RT/Handle.pm:814) [52450] [Tue Jan 28 19:16:07 2014] [debug]: Creating ACL... (/opt/rt4/sbin/../lib/RT/Handle.pm:1017) [52450] [Tue Jan 28 19:16:07 2014] [critical]: Can't use an undefined value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line 22, $handle line 1. (/opt/rt4/sbin/../lib/RT.pm:393) Can't use an undefined value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line 22, $handle line 1. -Original Message- From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Jonah Hirsch Sent: Tuesday, January 28, 2014 11:59 AM To: Alex Vandiver; rt-users@lists.bestpractical.com Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2 I get nothing in logs with debugging on before the system locks up. The issue still occurs after disabling all extensions but RT::Authen::ExternalAuth. -Original Message- From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Alex Vandiver Sent: Tuesday, January 28, 2014 11:45 AM To: rt-users@lists.bestpractical.com Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2 On Tue, 2014-01-28 at 16:23 +, Jonah Hirsch wrote: It seems to occur for any ticket. I tried it on our most innocuous queue and it still occurred. Removed all people from the ticket, no special things in the ticket, and it still causes its apache process to lock up the system. Do you have any extensions or customizations installed? Are there any errors in your error logs? - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
On Tue, 2014-01-28 at 19:18 +, Jonah Hirsch wrote: I just remembered that when upgrading I had some database upgrade errors. I checked the upgrade history in RT and saw that it failed upgrading the DB schema from 4.1.12 on. When I try to run /opt/rt4/sbin/rt-setup-database --action insert --datafile content From the /rt-4.2.2/etc/upgrade/4.1.12 directory I get the following including an error: The right way to try to resume the upgrade is /opt/rt4/sbin/rt-setup-database --action upgrade ...and enter 4.1.12 as the starting version. Now inserting data. [52450] [Tue Jan 28 19:16:07 2014] [debug]: Going to load 'content' data file (/opt/rt4/sbin/../lib/RT/Handle.pm:814) [52450] [Tue Jan 28 19:16:07 2014] [debug]: Creating ACL... (/opt/rt4/sbin/../lib/RT/Handle.pm:1017) [52450] [Tue Jan 28 19:16:07 2014] [critical]: Can't use an undefined value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line 22, $handle line 1. (/opt/rt4/sbin/../lib/RT.pm:393) Can't use an undefined value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line 22, $handle line 1. RT doesn't ship a System_Vendor.pm, so color me extremely suspicious. - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
I got the upgrades to run successfully, but I still get the memory issue when
merging tickets.
I noticed that System_Vendor.pm wasn't a stock RT file either, but I'm not sure
where it came from. Here's its contents:
package RT::System;
use strict;
no warnings qw(redefine);
sub AvailableRights {
my $self = shift;
my $queue = RT::Queue-new(RT-SystemUser);
my $group = RT::Group-new(RT-SystemUser);
my $cf= RT::CustomField-new(RT-SystemUser);
my $class = RT::Class-new(RT-SystemUser);
#my $type = RTx::AssetTracker::Type-new(RT-SystemUser);
my $qr = $queue-AvailableRights();
my $gr = $group-AvailableRights();
my $cr = $cf-AvailableRights();
my $clr = $class-AvailableRights();
#my $tr = $type-AvailableRights();
# Build a merged list of all system wide rights, queue rights and group
rights.
my %rights = (%{$RT::System::RIGHTS}, %{$gr}, %{$qr}, %{$cr}, %{$clr});#,
%$tr);
delete $rights{ExecuteCode} if RT-Config-Get('DisallowExecuteCode');
return(\%rights);
}
sub RightCategories {
my $self = shift;
my $queue = RT::Queue-new(RT-SystemUser);
my $group = RT::Group-new(RT-SystemUser);
my $cf= RT::CustomField-new(RT-SystemUser);
my $class = RT::Class-new(RT-SystemUser);
#my $type = RTx::AssetTracker::Type-new(RT-SystemUser);
my $qr = $queue-RightCategories();
my $gr = $group-RightCategories();
my $cr = $cf-RightCategories();
my $clr = $class-RightCategories();
# my $tr = $type-RightCategories();
# Build a merged list of all system wide rights, queue rights and group
rights.
my %rights = (%{$RT::System::RIGHT_CATEGORIES}, %{$gr}, %{$qr}, %{$cr},
%{$clr});#, %$tr);
return(\%rights);
}
1;
-Original Message-
From: rt-users-boun...@lists.bestpractical.com
[mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Alex Vandiver
Sent: Tuesday, January 28, 2014 12:41 PM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
On Tue, 2014-01-28 at 19:18 +, Jonah Hirsch wrote:
I just remembered that when upgrading I had some database upgrade
errors. I checked the upgrade history in RT and saw that it failed
upgrading the DB schema from 4.1.12 on. When I try to run
/opt/rt4/sbin/rt-setup-database --action insert --datafile content
From the /rt-4.2.2/etc/upgrade/4.1.12 directory I get the following
including an error:
The right way to try to resume the upgrade is
/opt/rt4/sbin/rt-setup-database --action upgrade
...and enter 4.1.12 as the starting version.
Now inserting data.
[52450] [Tue Jan 28 19:16:07 2014] [debug]: Going to load 'content'
data file (/opt/rt4/sbin/../lib/RT/Handle.pm:814)
[52450] [Tue Jan 28 19:16:07 2014] [debug]: Creating ACL...
(/opt/rt4/sbin/../lib/RT/Handle.pm:1017)
[52450] [Tue Jan 28 19:16:07 2014] [critical]: Can't use an undefined
value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line
22, $handle line 1. (/opt/rt4/sbin/../lib/RT.pm:393) Can't use an undefined
value as a HASH reference at /opt/rt4/sbin/../lib/RT/System_Vendor.pm line
22, $handle line 1.
RT doesn't ship a System_Vendor.pm, so color me extremely suspicious.
- Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
On Tue, 2014-01-28 at 21:12 +, Jonah Hirsch wrote: I got the upgrades to run successfully, but I still get the memory issue when merging tickets. I noticed that System_Vendor.pm wasn't a stock RT file either, but I'm not sure where it came from. Here's its contents: That would be the third-party RTx::AssetTracker extension -- which as fas as I'm aware does not function on RT 4.2. If it's installed into /opt/rt4/lib/, it's also installed incorrectly, which will make removing it somewhat more difficult -- you should remove all *_Vendor.pm, *_Overlay.pm, and *_Local.pm files from /opt/rt4/lib/ - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
Yeah I'd removed AssetTracker a while ago. Looks like I missed the _Vendor files since that's all that remained (CustomFields, CustomField, System, and Ticket _Vendor.pm files). Removing those files has fixed the issue. Thanks for helping me track this down! -Original Message- From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Alex Vandiver Sent: Tuesday, January 28, 2014 2:28 PM To: rt-users@lists.bestpractical.com Subject: Re: [rt-users] Memory leak when merging tickets, RT 4.2.2 On Tue, 2014-01-28 at 21:12 +, Jonah Hirsch wrote: I got the upgrades to run successfully, but I still get the memory issue when merging tickets. I noticed that System_Vendor.pm wasn't a stock RT file either, but I'm not sure where it came from. Here's its contents: That would be the third-party RTx::AssetTracker extension -- which as fas as I'm aware does not function on RT 4.2. If it's installed into /opt/rt4/lib/, it's also installed incorrectly, which will make removing it somewhat more difficult -- you should remove all *_Vendor.pm, *_Overlay.pm, and *_Local.pm files from /opt/rt4/lib/ - Alex
Re: [rt-users] Memory leak when merging tickets, RT 4.2.2
On Fri, 2014-01-24 at 17:43 +, Jonah Hirsch wrote: I just upgraded to RT 4.2.2 yesterday, and whenever a user merges a ticket, the responsible apache processes maxes out memory usage on the system and the system slows to a crawl until the OS kills the process, since the system is out of memory. Is this a known issue? Anyone have any thoughts on what I can do to try to solve this problem? This is the first I've heard of it. Does it occur for any ticket merge? Can you replicate it in a clean RT 4.2.2? - Alex
[rt-users] Memory leak when merging tickets, RT 4.2.2
I just upgraded to RT 4.2.2 yesterday, and whenever a user merges a ticket, the responsible apache processes maxes out memory usage on the system and the system slows to a crawl until the OS kills the process, since the system is out of memory. Is this a known issue? Anyone have any thoughts on what I can do to try to solve this problem? Jonah Hirsch
