Re: [Catalyst] Sending 3000 emails.

[Andrew]

Hi Len and Darren,
Atm I'm using Email::Stuffer to send email from within a perl script, in the
Controller folder of my Catalyst app.
I'm assuming this results in it being queued to be sent by the local SMTP
server.
My VPS server has Cpanel, and uses Exim, which I'm guessing is what the
scripts are using to send the email.

As noted, it all works until about 1020 emails in, at which point the
Catalyst app crashes (although this has only happened once - I do not wish
to try again and risk another crash). I am using Email::Stuffer in a for
loop, iterating over an array of 3000 email addresses. For my next step I
will alter my code to only attempt to send to 500 email addresses at any one
time, since below 1000 everything seems to work fine.

You seem to be mentioning using a background process, Darren. Is Exim
already the background process that sends email?

If the long term solution is to use mailgun or sendgrid - before I delve
into all their documentation - do you lot have any tips I should consider,
on how to integrate them with a Catalyst app, or is it all pretty straight
forward once I read the documentation?





- Original Message -
From: "Darren Duncan" 
To: "The elegant MVC web framework" 
Sent: Monday, November 30, 2015 11:41 PM
Subject: Re: [Catalyst] Sending 3000 emails.


On 2015-11-30 9:29 AM, Andrew wrote:
> ...given that my attempt to send 3000 emails with nothing but my own VPS
server,
> and Cpan's Email::Stuffer, only sent around 1000, before the
> plack/fastcgi/catalyst app crashed could I have been running into a
similar
> limitation? I'm now thinking of altering my code to split the attempts to
send
> email into six batches of 500 emails, as a short term fix, until I've had
time
> to research which transactional email partner to use, and how their APIs
work,
> or integrate with Catalyst apps.

If you are trying to send the emails directly from the process serving your
web
API, then this is absolutely the wrong approach, and you should give up
immediately on trying to do that.  Your API process should just queue the
messages to be sent by a separate background process, and then return
immediately such as with a message "3000 messages queued to be sent soon,
click
here to check queue progress" or such.  Sending in-process may be reasonable
for
a single email, eg email a copy of the form the user just submitted, but
that's
it. -- Darren Duncan


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Sending 3000 emails.

[Lance A. Brown]

On 12/2/2015 10:10 AM, Andrew wrote:
> As noted, it all works until about 1020 emails in, at which point the
> Catalyst app crashes (although this has only happened once - I do not wish
> to try again and risk another crash). I am using Email::Stuffer in a for
> loop, iterating over an array of 3000 email addresses. For my next step I
> will alter my code to only attempt to send to 500 email addresses at any one
> time, since below 1000 everything seems to work fine.

Are you sending those emails as 3,000 individual messages?  Are you
keeping the Email::Stuffer objects around after doing ->send on them?
With your Catalyst app crashing I wonder if you aren't hitting a memory
resource limit or memory leak bug.

--[Lance]


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Sending 3000 emails.

[Len Jaffe]

On Wed, Dec 2, 2015 at 10:10 AM, Andrew 
wrote:

> Hi Len and Darren,
> Atm I'm using Email::Stuffer to send email from within a perl script, in
> the
> Controller folder of my Catalyst app.
> I'm assuming this results in it being queued to be sent by the local SMTP
> server.
> My VPS server has Cpanel, and uses Exim, which I'm guessing is what the
> scripts are using to send the email.
>

According to the docs
http://search.cpan.org/~rjbs/Email-Stuffer-0.012/lib/Email/Stuffer.pm
unless you explicitly set the transport, Email::Stuffer defers to
Email::Sender for the default transport,
which appears to be sendmail, which means you'll be handing it off locally
for delivery.


As noted, it all works until about 1020 emails in, at which point the
> Catalyst app crashes (although this has only happened once - I do not wish
> to try again and risk another crash). I am using Email::Stuffer in a for
> loop, iterating over an array of 3000 email addresses. For my next step I
> will alter my code to only attempt to send to 500 email addresses at any
> one
> time, since below 1000 everything seems to work fine.
>

If you can consistently to 500, but 1000 consistently crashes you, and you
don't want to
spend the time on forensics, then just go with 500. But be aware that if
you're not freeing
memory, then it's possible to hit the resource usage limits if e.g. you
start sending a bigger message.

You seem to be mentioning using a background process, Darren. Is Exim
> already the background process that sends email?
>

It is a web service trope now that one does not send any email directly as
the result of a browser interaction,
but rather one notes that an email needs to be sent (queues a job request),
and a demon or cron job (a queue
processor) creates ans sends the mail. Thus the web infrastructure can
finish serving the web request sooner,
which is the greatest good at web scale.


If the long term solution is to use mailgun or sendgrid - before I delve
> into all their documentation - do you lot have any tips I should consider,
> on how to integrate them with a Catalyst app, or is it all pretty straight
> forward once I read the documentation?

I would not explore them unless:
1) you start sending a lot more mail.
2) you start ending up in spam control black lists.

As for tips, my usage of sendmail involves having my local sendmail (I use
postfix) hand off to
sendgrid for delivery.  I'd like to make use of their
thousand-addresses-at-a-time capabilities, but
my client doesn't feel the need to optimize their delivery processes yet.



Len.



> - Original Message -
> From: "Darren Duncan" 
> To: "The elegant MVC web framework" 
> Sent: Monday, November 30, 2015 11:41 PM
> Subject: Re: [Catalyst] Sending 3000 emails.
>
>
> On 2015-11-30 9:29 AM, Andrew wrote:
> > ...given that my attempt to send 3000 emails with nothing but my own VPS
> server,
> > and Cpan's Email::Stuffer, only sent around 1000, before the
> > plack/fastcgi/catalyst app crashed could I have been running into a
> similar
> > limitation? I'm now thinking of altering my code to split the attempts to
> send
> > email into six batches of 500 emails, as a short term fix, until I've had
> time
> > to research which transactional email partner to use, and how their APIs
> work,
> > or integrate with Catalyst apps.
>
> If you are trying to send the emails directly from the process serving your
> web
> API, then this is absolutely the wrong approach, and you should give up
> immediately on trying to do that.  Your API process should just queue the
> messages to be sent by a separate background process, and then return
> immediately such as with a message "3000 messages queued to be sent soon,
> click
> here to check queue progress" or such.  Sending in-process may be
> reasonable
> for
> a single email, eg email a copy of the form the user just submitted, but
> that's
> it. -- Darren Duncan
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>



-- 
Len Jaffe - Information Technology Smoke Jumper - lenja...@jaffesystems.com
614-404-4214@LenJaffe 
www.lenjaffe.com
Host of Code Jam Columbus   -
@CodeJamCMH 
Curator of Advent Planet  - An
Aggregation of Online Advent Calendars.

Re: [Catalyst] Sending 3000 emails.

[Len Jaffe]

On Wed, Dec 2, 2015 at 10:59 AM, Lance A. Brown 
wrote:

> On 12/2/2015 10:10 AM, Andrew wrote:
> > As noted, it all works until about 1020 emails in, at which point the
> > Catalyst app crashes (although this has only happened once - I do not
> wish
> > to try again and risk another crash). I am using Email::Stuffer in a for
> > loop, iterating over an array of 3000 email addresses. For my next step I
> > will alter my code to only attempt to send to 500 email addresses at any
> one
> > time, since below 1000 everything seems to work fine.
>
> Are you sending those emails as 3,000 individual messages?  Are you
> keeping the Email::Stuffer objects around after doing ->send on them?
> With your Catalyst app crashing I wonder if you aren't hitting a memory
> resource limit or memory leak bug.
>

copious logging will be your friend.
also, strace and Devel::NYTProf output might shed interesting light.







>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>



-- 
Len Jaffe - Information Technology Smoke Jumper - lenja...@jaffesystems.com
614-404-4214@LenJaffe 
www.lenjaffe.com
Host of Code Jam Columbus   -
@CodeJamCMH 
Curator of Advent Planet  - An
Aggregation of Online Advent Calendars.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/