Thanks, that clears some things up. I just ran a sizeable test and sent
~4000 messages to my James server. The spool indeed got filled up but
went down in size pretty quickly. The outgoing messages folder then got
filled up accordingly and is slowly (very slowly) processing messages.
It took about 10 minutes for the spool to completely empty but it's
taking longer for the outgoing messages to go down. I already have the
number of delivery threads set to 100. I can't imagine that it would
need more, right?
As for my the two Redirect mailets I've written. The first one,
BoltNoteRedirect, takes the username of a user from a header that's
attached to the original email, looks up the user's email from the James
userstore (which is in a SQL database), strips the username header,
replaces the To: field, which is in the form of [EMAIL PROTECTED] , with
a valid email address from the userstore's forwardDestination column
([EMAIL PROTECTED] for example), destroys the original email and puts the
new one on the spool. So, there is overhead for forwarding the email on
to aol, hotmail, yahoo or whatever domain is involved. Could this be my
bottleneck with respect to outgoing messages? The second mailet does
something similar except it knows the email address of the user ahead of
time so it doesn't need to look up the user's email address from the
user store.
On Thu, 2005-03-31 at 01:49 +0200, [EMAIL PROTECTED] wrote:
> > Thanks for the reply. I've included the <spoolmanager>
> > element of my config as well as the thread manager element.
> > Please let me know if there's anything else I need to
> > include. As for the outgoing mail folder, I can stop the
> > cron job from cleaning out the messages but how are messages
> > that get stuck in /var/mail/outgoing processed?
>
> The delivery threads will grab messages to be delivered from the outgoing
> spool but will not try too often the ones that are being delayed for the
> retries. The <delayTime> configurations allow you to change those retry
> times.
> I'm using a "db" spool repositories and not "file" so I don't know if your
> CPU time can be a not so performant file repository access.
> I see you have a DB set up for the "bolt" operations: why don't you try to
> move your spool to db? It should handle thousands of messages with no
> problems.
>
> > If a message
> > has an invalid email or can't be send for some reason or
> > another won't it remain in the outgoing folder until the
> > retries timeout?
>
> Yes. At the end of the retry count they will be bounced and deleted.
> Are you getting bounces?
>
> > Does not that compete with new messages
> > that are valid and waiting to be sent?
>
> In fact no.
>
> > I just did a count of
> > the messages in my spool and outgoing folders and this is what I see:
> > [EMAIL PROTECTED] find spool | wc -l
> > 1381
> > [EMAIL PROTECTED] find outgoing | wc -l
> > 2323
>
> It seems to me *strange" 1381 messages in spool with 50 spool threads!
> You told about 5000 messages per day so it seems that the latest 6 hour
> messages are waiting in spool and that 50 spool threads are not able to move
> them to the appropriate repository or to deliver them remotely.
>
> This could happen only if many of this messages go to the "bolt" processor
> and if the 2 mailet are doing something really wrong and taking too much to
> handle the messages.
> What does these 2 BoltNotesRedirect / BoltAutomatedRedirect do??
> Are you sure you are not creating mail-loops?
>
> > My <spoolmanager> and <thread-manager> elements are below.
> > I've changed the emails, database login/passwords and the
> > header names for privacy but everything else is exactly what
> > I'm using in production. Oh yeah, I'm using James 2.2.0.
> >
> > <spoolmanager>
> > <!-- Number of spool threads -->
> > <threads> 50 </threads>
> >
> > <!-- Set the Java packages from which to load mailets
> > and matchers
> > -->
> > <mailetpackages>
> >
> > <mailetpackage>org.apache.james.transport.mailets</mailetpackage>
> > <mailetpackage>com.bolt.james.mailets</mailetpackage>
> >
> > <mailetpackage>com.bolt.james.mailets.automated</mailetpackage>
> > <mailetpackage>com.bolt.james.mailets.bolt</mailetpackage>
> > <mailetpackage>com.bolt.james.mailets.notes</mailetpackage>
> > </mailetpackages>
> > <matcherpackages>
> >
> > <matcherpackage>org.apache.james.transport.matchers</matcherpackage>
> > </matcherpackages>
> >
> > <processor name="root">
> >
> > <!-- Checks that the email Sender is associated with
> > a valid domain. -->
> > <!-- Useful for detecting and eliminating spam. -->
> > <!-- For this block to function, the spam processor
> > must be configured. -->
> > <!--
> > <mailet
> > match="SenderInFakeDomain=64.55.105.9,64.94.110.11,194.205.62.
> 122,194.205.62.62,195.7.77.20,206.253.214.102,212.181.91.6,219.88.106.80,194
> .205.62.42,216.35.187.246,203.119.4.6" class="ToProcessor">
> > <processor> spam </processor>
> > </mailet>
> > -->
> >
> > <!-- Important check to avoid looping -->
> > <mailet match="RelayLimit=30" class="Null"/>
> >
> > <!--
> > <mailet match="All" class="XMLVirtualUserTable">
> > <!- 1:1 mapping ->
> > <mapping>[EMAIL PROTECTED]@mordor</mapping>
> > <!- 1:n mapping ->
> > <mapping>[EMAIL PROTECTED]
> > [EMAIL PROTECTED];radigast;gandalf</mapping>
> > <!- DSN mapping ->
> > <mapping>[EMAIL PROTECTED]:550 Requested action not
> > taken: no such user here</mapping>
> > <!- regex based mapping ->
> > <mapping>[EMAIL PROTECTED]:(.*)@osgilliath:[EMAIL PROTECTED]
> > tirith</mapping>
> > <!- both standard and regex mapping ->
> > <mapping>[EMAIL PROTECTED]@mordor;regex:ring@(.*):ring@
> > ${1}</mapping>
> > <!- conditional regex mapping example ->
> > <mapping>[EMAIL PROTECTED]:(.*)[EMAIL PROTECTED]:${1}-
> > [EMAIL PROTECTED];
> >
> > regex:(.*)[EMAIL PROTECTED]:${1}- [EMAIL PROTECTED]
> > </mapping>
> > </mailet>
> > -->
> >
> > <!-- White List:
> > If you use block lists, you will probably want to check
> > for known permitted senders. This is particularly true
> > if you use more aggressive block lists, such as SPEWS,
> > that are prone to block entire subnets without regard
> > for non-spamming senders.
> > -->
> >
> > <!-- People on this list agree to pay a penalty if
> > they send spam -->
> > <mailet match="InSpammerBlacklist=query.bondedsender.org"
> > class="ToProcessor">
> > <processor> remotedelivery </processor>
> > </mailet>
> >
> > <!-- End of White List -->
> >
> > <!-- Check for delivery from a known spam server -->
> > <!-- This set of matchers/mailets redirect all
> > emails from known -->
> > <!-- black holes, open relays, and spam servers to
> > the spam processor -->
> > <!-- For this set to function properly, the spam
> > processor must be configured. -->
> > <mailet match="InSpammerBlacklist=dnsbl.njabl.org"
> > class="ToProcessor">
> > <processor> spam </processor>
> > <notice>550 Requested action not taken: rejected -
> > see http://njabl.org/ </notice>
> > </mailet>
> >
> > <mailet match="InSpammerBlacklist=relays.ordb.org"
> > class="ToProcessor">
> > <processor> spam </processor>
> > <notice>550 Requested action not taken: rejected -
> > see http://www.ordb.org/ </notice>
> > </mailet>
> >
> > <!-- Sample matching to kill a message (send to Null) -->
> > <!--
> > <mailet match="[EMAIL PROTECTED]" class="Null"/>
> > -->
> >
> > <mailet match="[EMAIL PROTECTED]"
> > class="Null"/>
> >
> > <mailet match="HasHeader=X-Some-Header-1" class="ToProcessor">
> > <processor>bolt</processor>
> > </mailet>
> >
> > <mailet match="HasHeader=X-Some-Header-2" class="ToProcessor">
> > <processor>bolt</processor>
> > </mailet>
> >
> > <mailet match="All" class="ToProcessor">
> > <processor> remotedelivery </processor>
> > </mailet>
> > </processor>
> >
> > <processor name="bolt">
> >
> > <mailet match="HasHeader=X-Some-Header-1"
> > class="BoltNotesRedirect">
> > <static>false</static>
> > <inline>unaltered</inline>
> > <attachment>none</attachment>
> > <passThrough>false</passThrough>
> > <debug>false</debug>
> > <driver>net.sourceforge.jtds.jdbc.Driver</driver>
> > <url>jdbc:jtds:sqlserver://DBSERVER:6969/DB_NAME</url>
> > <user>db_login</user>
> > <password>db_password</password>
> > <preparedStatement>select email from UserEmail
> > where userId = (?)</preparedStatement>
> > <headers>X-Some-Header-1</headers>
> > </mailet>
> >
> > <mailet match="HasHeader=X-Some-Header-2"
> > class="BoltAutomatedRedirect">
> > <static>false</static>
> > <inline>unaltered</inline>
> > <attachment>none</attachment>
> > <passThrough>false</passThrough>
> > <debug>false</debug>
> > <headers>X-Some-Header-2,X-Some-Header-3</headers>
> > </mailet>
> >
> > </processor>
> >
> > <processor name="remotedelivery">
> >
> > <!-- This is an anti-relay matcher/mailet combination -->
> > <!-- -->
> > <!-- Emails sent from servers not in the network
> > list are -->
> > <!-- rejected as spam. This is one method of preventing your
> > -->
> > <!-- server from being used as an open relay. Make
> > sure you understand -->
> > <!-- how to prevent your server from becoming an
> > open relay before -->
> > <!-- changing this configuration. See also
> > <authorizedAddresses> in SMTP Server -->
> > <!-- -->
> > <!-- This matcher/mailet combination must come after
> > local delivery has -->
> > <!-- been performed. Otherwise local users will not
> > be able to receive -->
> > <!-- email from senders not in this remote address list. -->
> > <!-- -->
> > <!-- If you are using this matcher/mailet you will
> > probably want to -->
> > <!-- update the configuration to include your own
> > network/addresses. The -->
> > <!-- matcher can be configured with a comma
> > separated list of IP addresses -->
> > <!-- wildcarded IP subnets, and wildcarded hostname
> > subnets. --
> > >
> > <!-- e.g. "RemoteAddrNotInNetwork=127.0.0.1,
> > abc.de.*, 192.168.0.*" -->
> > <!-- -->
> > <!-- If you are using SMTP authentication then you
> > can (and generally -->
> > <!-- should) disable this matcher/mailet pair. -->
> >
> > <!-- Deleting as this is not needed. The definition
> > below would mean mail from yahoo.com mail servers would not
> > be able to get through. -->
> > <!--
> > <mailet match="RemoteAddrNotInNetwork=127.0.0.1, bolt.com"
> > class="ToProcessor">
> > <processor> relay-denied </processor>
> > <notice>550 - Requested action not taken:
> > relaying denied</notice>
> > </mailet>
> > -->
> >
> > <!-- Attempt remote delivery using the specified
> > repository for the spool, -->
> > <!-- using delay time to retry delivery and the
> > maximum number of retries -->
> > <mailet match="All" class="RemoteDelivery">
> > <outgoing> file://var/mail/outgoing/ </outgoing>
> > <!-- alternative database repository example below -->
> > <!--
> > <outgoing> db://maildb/spool/outgoing </outgoing>
> > -->
> >
> > <!-- Delivery Schedule based upon RFC 2821, 4.5.4.1 -->
> > <!-- 5 day retry period, with 4 attempts in the first
> > hour, two more within the first 6 hours, and then
> > every 6 hours for the rest of the period. -->
> > <delayTime> 5 minutes </delayTime>
> > <delayTime> 10 minutes </delayTime>
> > <delayTime> 20 minutes </delayTime>
> > <delayTime> 45 minutes </delayTime>
> > <delayTime> 2 hours </delayTime>
> > <delayTime> 3 hours </delayTime>
> > <delayTime> 6 hours </delayTime>
> > <maxRetries> 25 </maxRetries>
> >
> > <!-- The number of threads that should be
> > trying to deliver outgoing messages -->
> > <deliveryThreads> 100 </deliveryThreads>
> >
> > <!-- If false the message will not be sent to
> > given server if any recipients fail -->
> > <sendpartial>false</sendpartial>
> >
> > <!-- A single mail server to deliver all
> > outgoing messages. -->
> > <!-- This is useful if this server is a backup
> > or failover machine, -->
> > <!-- or if you want all messages to be routed
> > through a particular mail server, -->
> > <!-- regardless of the email addresses
> > specified in the message -->
> > <!-- -->
> > <!-- The gateway element specifies the gateway
> > SMTP server name. -->
> > <!-- If your gateway mail server is listening
> > on a port other than 25, -->
> > <!-- you can set James to connect to it on that
> > port using the gatewayPort -->
> > <!-- element. -->
> > <!-- Although normally multiple addresses are
> > implemented through proper -->
> > <!-- DNS configuration, the RemoteDelivery mail
> > does allow specifying -->
> > <!-- multiple gateway elements, each of which
> > may also have a port -->
> > <!-- e.g., mygateway:2525 -->
> > <!-- the gatewayPort element is used as a default -->
> > <!--
> > <gateway> otherserver.mydomain.com </gateway>
> > <gatewayPort>25</gatewayPort>
> > -->
> > </mailet>
> >
> > </processor>
> >
> >
> > <!-- The error processor is required. James may
> > internally set emails to the -->
> > <!-- error state. The error processor is generally
> > invoked when there is an -->
> > <!-- unexpected error either in the mailet chain or
> > internal to James. -->
> > <!-- -->
> > <!-- By default configuration all email that generates
> > an error in placed in -->
> > <!-- an error repository. -->
> > <processor name="error">
> > <!-- If you want to notify the sender their message generated
> > an error, uncomment this -->
> > <mailet match="All" class="Bounce"/>
> >
> > <!-- If you want to notify the postmaster that a
> > message generated an error, uncomment this -->
> > <!--
> > <mailet match="All" class="NotifyPostmaster"/>
> > -->
> >
> > <!-- Logs any messages to the repository specified -->
> > <!-- <mailet match="All" class="ToRepository">-->
> > <!-- <repositoryPath>
> > file://var/mail/error/</repositoryPath>-->
> > <!-- An alternative database repository example
> > follows. -->
> > <!--
> > <repositoryPath> db://maildb/deadletter/error
> > </repositoryPath>
> > -->
> > <!-- </mailet>-->
> >
> > <!-- Delete all error messages instead of spooling
> > them into var/mail/error (Ricardo 1/14/2005) -->
> > <mailet match="All" class="Null"/>
> >
> > </processor>
> >
> > <!-- Processor CONFIGURATION SAMPLE: spam is a sample
> > custom processor for handling -->
> > <!-- spam. -->
> > <!-- You can either log these, bounce these, or just
> > ignore them.
> > -->
> > <processor name="spam">
> > <!-- To destroy all messages, uncomment this
> > matcher/mailet configuration -->
> > <!--
> > <mailet match="All" class="Null"/>
> > -->
> >
> > <!-- To notify the sender their message was marked
> > as spam, uncomment this matcher/mailet configuration -->
> > <!--
> > <mailet match="All" class="Bounce"/>
> > -->
> >
> > <!-- To notify the postmaster that a message was
> > marked as spam, uncomment this matcher/mailet configuration -->
> > <!--
> > <mailet match="All" class="NotifyPostmaster"/>
> > -->
> >
> > <!-- To log the message to a repository, this
> > matcher/mailet configuration should be uncommented. -->
> > <!-- This is the default configuration. -->
> > <!-- <mailet match="All" class="ToRepository">-->
> > <!--
> > <repositoryPath>file://var/mail/spam/</repositoryPath>--
> > >
> >
> > <!-- Changing the repositoryPath, as in this
> > commented out example, will -->
> > <!-- cause the mails to be stored in a database
> > repository.
> > -->
> > <!-- Please note that only one repositoryPath
> > element can be present for the mailet -->
> > <!-- configuration. -->
> > <!--
> > <repositoryPath> db://maildb/deadletter/spam
> > </repositoryPath>
> > -->
> > <!-- </mailet>-->
> >
> > <!-- Delete all spam messages instead of spooling
> > them into var/mail/spam (Ricardo 1/14/2005) -->
> > <mailet match="All" class="Null"/>
> >
> > </processor>
> >
> > <!-- This processor handles messages that are for local
> > domains, where the user is unknown -->
> > <processor name="local-address-error">
> > <!-- To notify the sender the address was invalid,
> > uncomment this matcher/mailet configuration -->
> > <!-- The original message is not attached to keep
> > the bounce processor from deliverying spam -->
> > <mailet match="All" class="Bounce">
> > <attachment>none</attachment>
> > </mailet>
> >
> > <!-- To notify the postmaster that a message had an
> > invalid address, uncomment this matcher/mailet configuration -->
> > <!--
> > <mailet match="All" class="NotifyPostmaster"/>
> > -->
> >
> > <!-- <mailet match="All" class="ToRepository">-->
> > <!-- <repositoryPath> file://var/mail/address-
> > error/</repositoryPath>-->
> > <!-- An alternative database repository example
> > follows. -->
> > <!--
> > <repositoryPath> db://maildb/deadletter/address-error
> > </repositoryPath>
> > -->
> > <!-- </mailet>-->
> >
> > <!-- Delete all local-address-error messages
> > instead of spooling them into var/mail (Ricardo 1/14/2005) -->
> > <mailet match="All" class="Null"/>
> >
> > </processor>
> >
> > <!-- This processor handles messages that are for
> > foreign domains, where relaying is denied -->
> > <!-- As of James v2.2, this processor can be deprecated
> > by using the <authorizedAddresses> tag
> > in the SMTP Server, and rejecting the message in
> > the protocol transaction. -->
> > <processor name="relay-denied">
> > <!-- To notify the sender the address was invalid,
> > uncomment this matcher/mailet configuration -->
> > <!-- The original message is not attached to keep
> > the bounce processor from deliverying spam -->
> > <mailet match="All" class="Bounce">
> > <attachment>none</attachment>
> > </mailet>
> >
> > <!-- To notify the postmaster that a relay request
> > was denied, uncomment this matcher/mailet configuration -->
> > <!--
> > <mailet match="All" class="NotifyPostmaster"/>
> > -->
> >
> > <!-- <mailet match="All" class="ToRepository">-->
> > <!-- <repositoryPath>file://var/mail/relay-
> > denied/</repositoryPath>-->
> > <!-- An alternative database repository example
> > follows. -->
> > <!--
> > <repositoryPath> db://maildb/deadletter/relay-denied
> > </repositoryPath>
> > -->
> > <!-- </mailet>-->
> >
> > <!-- Delete all relay-denied messages instead of
> > spooling them into var/mail/error (Ricardo 1/14/2005) -->
> > <mailet match="All" class="Null"/>
> >
> > </processor>
> > </spoolmanager>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
