mail loop problems: was: Re: [rt-users] RT/Apache suddenly hangs
Hi,
We have checked the problem more closesly, and the reason why our
RT/Apache server stops is due to different variants of mail loops.
We still have problems with other loop variants consuming all resources.
Extended mailgate errors leading to loops, spam going into loop. How can
we make mailgate a bit more smarter?
Does anyone have a mail loop detector in front of their RT installations.
We have a high severity now with our RT in production. The load is
increasing all the time.
Our commonly used cron-script removing some mail loops:
#!/site/perl-5.8.7/bin/perl
#
# Author: Petter Reinholdtsen
# Date:2006-01-13
# License: GNU Public License v2 or later
#
# Look at all tickets, and remove all queue addresses from requestor,
# cc and admincc. This reduces the amount of bounce emails sent to
# the RT admins.
use warnings;
use strict;
use Getopt::Std;
# Location of RT's libs and scripts
# Remember to change to correct path on current RT instance
use lib (/site/rt3/local/lib, /site/rt3/lib);
package RT;
use RT::Interface::CLI qw(CleanEnv);
use RT::Queue;
use RT::Queues;
use RT::Tickets;
my %opts;
Getopt::Std::getopts(dn, \%opts);
my $debug = $opts{'d'} || 0;
my $dryrun = $opts{'n'} || 0;
$| = 1;
# Find all queue addresses of enabled queues
my @queueaddrs =
(
# Aliases for e-mail lists are listed here:
'[EMAIL PROTECTED]',
);
my $ticketcount = 0;
my $starttime = time();
CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT
my $user = RT::User-new( $RT::SystemUser );
# Merge static list with dynamic list
@queueaddrs = (load_queue_addresses(), @queueaddrs);
# Loop over all addresses, remove them from the tickets were they are
# registered as watchers
for my $address( sort @queueaddrs ) {
print Removing address '$address' from all tickets\n if $debug;
my $tickets = new RT::Tickets($RT::SystemUser);
$tickets-FromSQL( Watcher.EmailAddress = '$address' );
while( my $ticket = $tickets-Next ) {
$ticketcount++;
my $id = $ticket-Id;
if ($dryrun) {
print Want to remove $address as watcher from ticket #$id\n;
} else {
$RT::Logger-info(Removed queue address $address as watcher
.
from ticket #$id);
$ticket-DeleteWatcher(Email = $address, Type = $_ )
for( qw(Cc AdminCc Requestor) );
}
}
}
my $duration = time() - $starttime;
$RT::Logger-info(Processing of $ticketcount tickets took $duration
seconds);
sub load_queue_addresses {
my $queues = new RT::Queues($RT::SystemUser);
$queues-LimitToEnabled();
my @queueaddrs;
foreach my $queue (@{$queues-ItemsArrayRef()}) {
for my $address ($queue-CorrespondAddress,
$queue-CommentAddress) {
next unless $user-LoadByEmail( $address );
push @queueaddrs, $address;
print Found queue address '$address'\n if $debug;
}
}
return @queueaddrs;
}
--
Tomas A. P. Olaj, email: [EMAIL PROTECTED], web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group
___
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
Community help: http://wiki.bestpractical.com
Commercial support: [EMAIL PROTECTED]
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
Re: mail loop problems: was: Re: [rt-users] RT/Apache suddenly hangs
Have you seen option RTAddressRegexp in the RT config?
Read http://wiki.bestpractical.com/?EmailInterface and
http://wiki.bestpractical.com/?RTAddressRegexp
On 2/7/07, Tomas Olaj [EMAIL PROTECTED] wrote:
Hi,
We have checked the problem more closesly, and the reason why our
RT/Apache server stops is due to different variants of mail loops.
We still have problems with other loop variants consuming all resources.
Extended mailgate errors leading to loops, spam going into loop. How can
we make mailgate a bit more smarter?
Does anyone have a mail loop detector in front of their RT installations.
We have a high severity now with our RT in production. The load is
increasing all the time.
Our commonly used cron-script removing some mail loops:
#!/site/perl-5.8.7/bin/perl
#
# Author: Petter Reinholdtsen
# Date:2006-01-13
# License: GNU Public License v2 or later
#
# Look at all tickets, and remove all queue addresses from requestor,
# cc and admincc. This reduces the amount of bounce emails sent to
# the RT admins.
use warnings;
use strict;
use Getopt::Std;
# Location of RT's libs and scripts
# Remember to change to correct path on current RT instance
use lib (/site/rt3/local/lib, /site/rt3/lib);
package RT;
use RT::Interface::CLI qw(CleanEnv);
use RT::Queue;
use RT::Queues;
use RT::Tickets;
my %opts;
Getopt::Std::getopts(dn, \%opts);
my $debug = $opts{'d'} || 0;
my $dryrun = $opts{'n'} || 0;
$| = 1;
# Find all queue addresses of enabled queues
my @queueaddrs =
(
# Aliases for e-mail lists are listed here:
'[EMAIL PROTECTED]',
);
my $ticketcount = 0;
my $starttime = time();
CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT
my $user = RT::User-new( $RT::SystemUser );
# Merge static list with dynamic list
@queueaddrs = (load_queue_addresses(), @queueaddrs);
# Loop over all addresses, remove them from the tickets were they are
# registered as watchers
for my $address( sort @queueaddrs ) {
print Removing address '$address' from all tickets\n if $debug;
my $tickets = new RT::Tickets($RT::SystemUser);
$tickets-FromSQL( Watcher.EmailAddress = '$address' );
while( my $ticket = $tickets-Next ) {
$ticketcount++;
my $id = $ticket-Id;
if ($dryrun) {
print Want to remove $address as watcher from ticket #$id\n;
} else {
$RT::Logger-info(Removed queue address $address as watcher
.
from ticket #$id);
$ticket-DeleteWatcher(Email = $address, Type = $_ )
for( qw(Cc AdminCc Requestor) );
}
}
}
my $duration = time() - $starttime;
$RT::Logger-info(Processing of $ticketcount tickets took $duration
seconds);
sub load_queue_addresses {
my $queues = new RT::Queues($RT::SystemUser);
$queues-LimitToEnabled();
my @queueaddrs;
foreach my $queue (@{$queues-ItemsArrayRef()}) {
for my $address ($queue-CorrespondAddress,
$queue-CommentAddress) {
next unless $user-LoadByEmail( $address );
push @queueaddrs, $address;
print Found queue address '$address'\n if $debug;
}
}
return @queueaddrs;
}
--
Tomas A. P. Olaj, email: [EMAIL PROTECTED], web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group
___
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
Community help: http://wiki.bestpractical.com
Commercial support: [EMAIL PROTECTED]
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
--
Best regards, Ruslan.
___
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
Community help: http://wiki.bestpractical.com
Commercial support: [EMAIL PROTECTED]
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] RT/Apache suddenly hangs
On the marvelous Fri, 2 Feb 2007, Torsten Brumm wrote kindly to me ... The only thing i found sometimes are messages: Mysql server has gone away, but this is causing not in all cases the problems. Nightly, without RT usage, the connections from FastCGI are timing out and don't connect automatically, but not during the business hours ;-( We are very satisfied with our Postgres installation. :) Our Postgres admins has created an administration framework used to administrate postgreSQL installations at The University of Oslo. This framework is a set of scripts, websides, C code, SQL definitions and standards that define storage,backups,maintenance and administration procedures. We can run this framework in a standalone server or in a SG-cluster from HP. Our RT production instance runs on a service guard two-node postgres cluster. Still faaar away from the sizes of all of our Oracle installations. ;) Thanks, Torsten, good to hear that I am not lonely about this problem. Cheers, Tomas -- Tomas A. P. Olaj, email: [EMAIL PROTECTED], web: folk.uio.no/tomaso University of Oslo / USIT (Center for Information Technology Services) System- and Application Management / Applications Management Group ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
[rt-users] RT/Apache suddenly hangs
System: === Red Hat Enterprise Linux WS release 3 (Taroon Update 8) RT 3.6.1 Apache v2.0.59 Perl 5.8.7 mod_fcgi-2.4.2 Postgres 8.1.4 Approximately 80.000 tickets. Problem: RT/Apache suddenly becomes unavailable/hangs (normaly once a day), and requires Apache restart so RT can work again. We are not sure what causes the problem, and if others have similar problems, we would be gladly to hear about it! List of processes running and load on server: = # ps aux | grep apache root 22066 0.0 0.0 7964 3360 ?SFeb01 0:08 /local/opt/apache2/bin/httpd -k start -DSSL nobody 13898 0.0 0.0 7964 3320 ?S03:59 0:00 /local/opt/apache2/bin/fcgi- -k start -DSSL nobody 21699 0.0 0.0 8216 3832 ?S14:02 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22628 0.0 0.0 8216 3840 ?S14:14 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22648 0.0 0.0 8216 3840 ?S14:15 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22650 0.0 0.0 8216 3820 ?S14:15 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22939 0.0 0.0 8252 3756 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22941 0.0 0.0 8216 3848 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22945 0.0 0.0 8216 3804 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22953 0.0 0.0 8216 3756 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22955 0.0 0.0 8216 3796 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22959 0.0 0.0 8216 3812 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22961 0.0 0.0 8216 3800 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22962 0.0 0.0 8216 3788 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22965 0.0 0.0 8236 3804 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22966 0.0 0.0 8216 3788 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22967 0.0 0.0 8216 3812 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23217 0.0 0.0 8228 3792 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23218 0.0 0.0 8228 3744 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23219 0.0 0.0 8244 3740 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23224 0.0 0.0 8232 3768 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23225 0.0 0.0 8216 3752 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23230 0.0 0.0 8228 3776 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23240 0.0 0.0 8220 3780 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23241 0.0 0.0 8220 3740 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23242 0.0 0.0 8248 3728 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23250 0.0 0.0 8216 3732 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23254 0.0 0.0 8216 3744 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23255 0.0 0.0 8216 3732 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23286 0.0 0.0 8216 3772 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23290 0.0 0.0 8216 3760 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23292 0.0 0.0 8248 3724 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23294 0.0 0.0 8216 3764 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23299 0.0 0.0 8108 3696 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23326 0.0 0.0 8108 3672 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23327 0.0 0.0 8108 3708 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23328 0.0 0.0 8216 3696 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23336 0.1 0.0 8248 3744 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23337 0.0 0.0 8108 3692 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23338 0.0 0.0 8108 3680 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23339 0.0 0.0 8108 3692 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23352 0.0 0.0
Re: [rt-users] RT/Apache suddenly hangs
Hi Tomas, yes, we have exactly the same under RHAS4 with Apache, FastCGI and mysql but we have to restart 4 times the day :-( Torsten 2007/2/2, Tomas Olaj [EMAIL PROTECTED]: System: === Red Hat Enterprise Linux WS release 3 (Taroon Update 8) RT 3.6.1 Apache v2.0.59 Perl 5.8.7 mod_fcgi-2.4.2 Postgres 8.1.4 Approximately 80.000 tickets. Problem: RT/Apache suddenly becomes unavailable/hangs (normaly once a day), and requires Apache restart so RT can work again. We are not sure what causes the problem, and if others have similar problems, we would be gladly to hear about it! List of processes running and load on server: = # ps aux | grep apache root 22066 0.0 0.0 7964 3360 ?SFeb01 0:08 /local/opt/apache2/bin/httpd -k start -DSSL nobody 13898 0.0 0.0 7964 3320 ?S03:59 0:00 /local/opt/apache2/bin/fcgi- -k start -DSSL nobody 21699 0.0 0.0 8216 3832 ?S14:02 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22628 0.0 0.0 8216 3840 ?S14:14 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22648 0.0 0.0 8216 3840 ?S14:15 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22650 0.0 0.0 8216 3820 ?S14:15 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22939 0.0 0.0 8252 3756 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22941 0.0 0.0 8216 3848 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22945 0.0 0.0 8216 3804 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22953 0.0 0.0 8216 3756 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22955 0.0 0.0 8216 3796 ?S14:18 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22959 0.0 0.0 8216 3812 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22961 0.0 0.0 8216 3800 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22962 0.0 0.0 8216 3788 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22965 0.0 0.0 8236 3804 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22966 0.0 0.0 8216 3788 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 22967 0.0 0.0 8216 3812 ?S14:19 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23217 0.0 0.0 8228 3792 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23218 0.0 0.0 8228 3744 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23219 0.0 0.0 8244 3740 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23224 0.0 0.0 8232 3768 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23225 0.0 0.0 8216 3752 ?S14:21 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23230 0.0 0.0 8228 3776 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23240 0.0 0.0 8220 3780 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23241 0.0 0.0 8220 3740 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23242 0.0 0.0 8248 3728 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23250 0.0 0.0 8216 3732 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23254 0.0 0.0 8216 3744 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23255 0.0 0.0 8216 3732 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23286 0.0 0.0 8216 3772 ?S14:22 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23290 0.0 0.0 8216 3760 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23292 0.0 0.0 8248 3724 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23294 0.0 0.0 8216 3764 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23299 0.0 0.0 8108 3696 ?S14:23 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23326 0.0 0.0 8108 3672 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23327 0.0 0.0 8108 3708 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23328 0.0 0.0 8216 3696 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23336 0.1 0.0 8248 3744 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23337 0.0 0.0 8108 3692 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start -DSSL nobody 23338 0.0 0.0 8108 3680 ?S14:25 0:00 /local/opt/apache2/bin/httpd -k start
Re: [rt-users] RT/Apache suddenly hangs
Don't know if this helps or not but do you have debug set in your RT_SiteConfig.pm file ? e.g. Set($LogToSyslog, 'debug'); Set($LogToScreen, 'error'); Set($LogToFile , 'debug'); Set($LogDir, '/opt/rt/var/log'); Set($LogToFileNamed , rt.log);#log to rt.log It has helped me out on several occasions! Alison -- ++ Alison Downie, Computing Officer School of Informatics, University of Edinburgh Room B21, 5 Forrest Hill, EDINBURGH, EH1 2QL Tel: 650 3095 ++ ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] RT/Apache suddenly hangs
The only thing i found sometimes are messages: Mysql server has gone away, but this is causing not in all cases the problems. Nightly, without RT usage, the connections from FastCGI are timing out and don't connect automatically, but not during the business hours ;-( 2007/2/2, Alison Downie [EMAIL PROTECTED]: Don't know if this helps or not but do you have debug set in your RT_SiteConfig.pm file ? e.g. Set($LogToSyslog, 'debug'); Set($LogToScreen, 'error'); Set($LogToFile , 'debug'); Set($LogDir, '/opt/rt/var/log'); Set($LogToFileNamed , rt.log);#log to rt.log It has helped me out on several occasions! Alison -- ++ Alison Downie, Computing Officer School of Informatics, University of Edinburgh Room B21, 5 Forrest Hill, EDINBURGH, EH1 2QL Tel: 650 3095 ++ ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com -- MFG Torsten Brumm http://www.torsten-brumm.de ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] RT/Apache suddenly hangs
On Fri, Feb 02, 2007 at 04:47:11PM +0100, Torsten Brumm wrote: The only thing i found sometimes are messages: Mysql server has gone away, but this is causing not in all cases the problems. Nightly, without RT usage, the connections from FastCGI are timing out and don't connect automatically, but not during the business hours ;-( If it fails during not-business-hours, that suggests the mysql morning bug which might be a reasonable google term. ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
