Re: [rt-users] Help - Deep Recursion on subroutine...
Postfix was definitely off, and nothing was injecting into the mailgateway, but it seems that the errors needed a few hours - most of a 24 hour day in fact - to go away. Unfortunately I wasn't paying too much attention, as I assumed that something was stuck in an internal loop :( I'm still working on the problem, but so far it seems that it may have been caused by an external mail loop (rt sending to an external user who was forwarding to another email address that was forwarding to RT). But I'm not sure. Still busy troubleshooting. I'll post success or failure of this testing phase to the list. Have also migrated from apache to nginx - if the problem seems to be resolved I will run with apache again for a while to ensure that the problem wasn't somehow related to the apache conf (but then, rt was running for a year with the apache conf so I don't think that will be the issue). Jason On Tue, Apr 30, 2013 at 12:19 AM, Thomas Sibley t...@bestpractical.comwrote: On 04/26/2013 10:28 PM, Jason Doller wrote: I've fixed and checked all the potential problems listed, and the support email address may have been the problem. However, I've turned off postfix, and I'm still getting the errors below. I realise I can probably clear a cache to stop this, but with postfix turned off, nothing is injecting new mails into the system, I'm concerned that this problem isn't going away. I've also (temporarily) turned off loop mail forwarding to owner... For those errors to pop up in your logs again, *something* is submitting mail to RT. Look at your access logs for hits to /REST/1.0/NoAuth/mail-gateway.
Re: [rt-users] Help - Deep Recursion on subroutine...
On 04/26/2013 10:28 PM, Jason Doller wrote: I've fixed and checked all the potential problems listed, and the support email address may have been the problem. However, I've turned off postfix, and I'm still getting the errors below. I realise I can probably clear a cache to stop this, but with postfix turned off, nothing is injecting new mails into the system, I'm concerned that this problem isn't going away. I've also (temporarily) turned off loop mail forwarding to owner... For those errors to pop up in your logs again, *something* is submitting mail to RT. Look at your access logs for hits to /REST/1.0/NoAuth/mail-gateway.
[rt-users] Help - Deep Recursion on subroutine...
Hi All I'm getting this error: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm in my logs, and I'm out of RAM. I'm also getting a lot of bounces: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) I've looked for loops in the RT_SiteConfig file (OwnerEmail) but see nothing. I've reinstalled the machine 4 times siince the problem started, on all combinations of: * RT 3.8.9 (original) * RT 4.0.11 * 512MB - 1GB RAM * CentOS 5.x - Centos 6.x And the problem persists. RT was originally running for about a year without a single problem or error on a CentOS 5 box with 512MB RAM. Problems started when I migrated to CentOS 6 (Same RT 3.8.9), and have persisted ever since. I'm not sure what else to check, where to look, or how to troubleshoot any further. Any help would be appreciated. Regards, Jason
Re: [rt-users] Help - Deep Recursion on subroutine...
On 04/26/2013 10:01 AM, Jason Doller wrote: I'm getting this error: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm in my logs, and I'm out of RAM. I'm also getting a lot of bounces: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) I've looked for loops in the RT_SiteConfig file (OwnerEmail) but see nothing. This implies a very deep mail loop that is getting larger and larger and hitting recursion limits when processing each successively larger bounce. RT should be logging at the [crit] level the message IDs. Can you find those and provide them? Your RT_SiteConfig.pm would be good to provide too. Folks can't help without these details.
Re: [rt-users] Help - Deep Recursion on subroutine...
Sorry for the oversight, RT_SiteConfig:: == Start == Set($rtname , IT Support); Set($Organization , obscured.com); Set($MinimumPasswordLength , 5); Set($Timezone , 'Africa/Johannesburg'); Set($HomepageComponents, [qw( QuickCreate Quicksearch MyAdminQueues MySupportQueues MyReminders RefreshHomepage Dashboards )]); Set($DatabaseType , 'mysql'); Set($DatabaseHost , 'localhost'); Set($DatabaseRTHost , 'localhost'); Set($DatabasePort , ''); Set($DatabaseUser , 'someuser'); Set($DatabasePassword , 'reallyreallystrongpassword'); Set($DatabaseName , 'rt4'); #Set($OwnerEmail , 'IT Support'); Set($OwnerEmail , 'ja...@obscured.com'); Set($LoopsToRTOwner , 1); Set($SendmailArguments , -oi -t -f support\@obscured.com); Set($MaxAttachmentSize , 1000); #Set($RTAddressRegexp , '(^rt\@obscured.com$)(.*\@obscured.co.za)'); Set($RTAddressRegexp , '.*\@obscured.co.za'); Set($CorrespondAddress , 'no-re...@obscured.co.za'); Set($CommentAddress , 'no-re...@obscured.co.za'); Set($UseFriendlyFromLine , 1); Set($FriendlyFromLineFormat , \%s\ %s); Set($UseFriendlyToLine , 1); Set($FriendlyToLineFormat, \%s Ticket #%s\:;); Set($NotifyActor, 0); Set($RecordOutgoingEmail, 1); Set($WebPath , /customerzone); Set($WebPort , 443); Set($WebBaseURL , https://obscured.com;); Set($WebURL , $WebBaseURL . $WebPath . /); Set($CanonicalizeRedirectURLs, 1); Set($MessageBoxWidth , 72); Set($MessageBoxWrap, HARD); Set($MaxInlineBody, 13456); Set($DefaultSummaryRows, 10); Set($OldestTransactionsFirst, '1'); Set($ShowTransactionImages, 1); Set($DateDayBeforeMonth , 0); Set($AmbiguousDayInPast , 1); Set($LogDir, '/var/log/rt4'); 1; == END == This is a sample of the email error I get (please let me know if you need more log info): [Fri Apr 26 20:17:12 2013] [critical]: RT Received mail ( 20130426001825.08c1521...@obscured.com ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Fri Apr 26 20:17:12 2013] [crit]: RT thinks this message may be a bounce (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:248) [Fri Apr 26 20:17:14 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Fri Apr 26 20:17:40 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) I am unsure how to get more information from the message id 20130426001825.08c1521...@obscured.com or whatever it is at that time. I don't find an equivalent message in themaillog. Thanks for the prompt response, Jason On Fri, Apr 26, 2013 at 9:12 PM, Thomas Sibley t...@bestpractical.comwrote: On 04/26/2013 10:01 AM, Jason Doller wrote: I'm getting this error: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm in my logs, and I'm out of RAM. I'm also getting a lot of bounces: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) I've looked for loops in the RT_SiteConfig file (OwnerEmail) but see nothing. This implies a very deep mail loop that is getting larger and larger and hitting recursion limits when processing each successively larger bounce. RT should be logging at the [crit] level the message IDs. Can you find those and provide them? Your RT_SiteConfig.pm would be good to provide too. Folks can't help without these details.
Re: [rt-users] Help - Deep Recursion on subroutine...
There are a number of possibilities given the config snippet below. Jason wrote: Set($SendmailArguments , -oi -t -f support\@obscured.com); Does supp...@obscured.com go into RT? -f sets the envelope-sender address which is where bounces go. If it goes into RT, that's potentially a mail loop. Set($MaxAttachmentSize , 1000); #Set($RTAddressRegexp , '(^rt\@obscured.com$)(.*\@obscured.co.za)'); Set($RTAddressRegexp , '.*\@obscured.co.za'); The commented out $RTAddressRegexp (which I suppose is commented out because it didn't work; the regex syntax is incorrect) contains r...@obscured.com, but the uncommented version of the option doesn't match that. Does r...@obscured.com still go into RT? If so, that could be the source of a loop if a ticket got a watcher of the rt@ address added. Try this instead: '^(rt@obscured\.com|.*@obscured\.co\.za)$' Escaping the @ isn't necessary in a single-quoted string, just double quoted to prevent interpolation. Additionally, the current $RTAddressRegexp should have the dots (.) in .co.za escaped with a backslash, though that's unlikely to cause a loop. Set($CorrespondAddress , 'no-re...@obscured.co.za'); Set($CommentAddress , 'no-re...@obscured.co.za'); These values are strange, since normally those addresses are intended to receive replies! However, they are unlikely to be a mail loop source since they will match your RTAddressRegexp. Thomas
Re: [rt-users] Help - Deep Recursion on subroutine...
I've fixed and checked all the potential problems listed, and the support email address may have been the problem. However, I've turned off postfix, and I'm still getting the errors below. I realise I can probably clear a cache to stop this, but with postfix turned off, nothing is injecting new mails into the system, I'm concerned that this problem isn't going away. I've also (temporarily) turned off loop mail forwarding to owner... Errors: [Sat Apr 27 05:21:10 2013] [critical]: RT Received mail ( 20130425182344.6aefa20...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:21:10 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Sat Apr 27 05:24:27 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:24:28 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:24:42 2013] [critical]: RT Received mail ( 20130426052216.69e6721...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:24:42 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Sat Apr 27 05:24:43 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:24:43 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:24:56 2013] [critical]: RT Received mail ( 20130425150731.6ef682d...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:24:57 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Sat Apr 27 05:25:44 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:25:44 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:25:57 2013] [critical]: RT Received mail ( 20130426105238.62c7721...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:25:58 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Sat Apr 27 05:26:19 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:26:19 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:26:28 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:26:29 2013] [warning]: Deep recursion on subroutine RT::I18N::SetMIMEEntityToEncoding at /opt/rt4/sbin/../lib/RT/I18N.pm line 210. (/opt/rt4/sbin/../lib/RT/I18N.pm:210) [Sat Apr 27 05:26:32 2013] [critical]: RT Received mail ( 20130426051826.676422f...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:26:32 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) [Sat Apr 27 05:26:43 2013] [critical]: RT Received mail ( 20130426045919.6d40e21...@ingress01.customermanager.co.za ) from itself. (/opt/rt4/sbin/../lib/RT/Interface/Email.pm:1846) [Sat Apr 27 05:26:43 2013] [error]: Could not record email: Message Bounced (/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75) Thanks for the help so far, Jason On Sat, Apr 27, 2013 at 1:35 AM, Thomas Sibley t...@bestpractical.comwrote: There are a number of possibilities given the config snippet below. Jason wrote: Set($SendmailArguments , -oi -t -f support\@obscured.com); Does supp...@obscured.com go into RT? -f sets the envelope-sender address which is where bounces go. If it goes into RT, that's potentially a mail loop. Set($MaxAttachmentSize , 1000); #Set($RTAddressRegexp , '(^rt\@obscured.com$)(.*\@obscured.co.za)'); Set($RTAddressRegexp , '.*\@obscured.co.za'); The commented out $RTAddressRegexp (which I suppose is commented out because it didn't work; the regex syntax is incorrect) contains r...@obscured.com, but the uncommented version of the option doesn't match that. Does r...@obscured.com still